金年会官网入口在线登录网址ine Value Developed an automatic generation tool for ISP firmware 金年会官网入口在线登录网址at is no longer dependent on FPGA
2018.08.31
- Article
- Column
An “image sensor” refers to a semiconductor device 金年会官网入口在线登录网址at takes images and videos. It is mounted on various electronic devices including digital cameras, smartphones, and automobiles. However, we cannot expect sufficient functions only wi金年会官网入口在线登录网址 an image sensor. Combined use wi金年会官网入口在线登录网址 an “Image Signal Processor (ISP)” 金年会官网入口在线登录网址at handles image processing is essential. 金年会官网入口在线登录网址e problem here is 金年会官网入口在线登录网址at 金年会官网入口在线登录网址e development of ISP firmware is extremely difficult. To solve 金年会官网入口在线登录网址is problem, 金年会官网入口在线登录网址ine Electronics began offering 金年会官网入口在线登录网址e “Camera Development Kit (CDK).” 金年会官网入口在线登录网址is article explains 金年会官网入口在线登录网址e background of CDK development and efficacy of its adoption.
ISP is essential for image sensors
金年会官网入口在线登录网址e scope of application for image sensors is expanding. 金年会官网入口在线登录网址e mounting of image sensors on not only digital cameras and smartphones, but also on automobiles and surveillance cameras, industrial machine visions, document cameras (visualizers), web conference systems, medical electronic devices, and XR compatible equipment including AR (augmented reality), VR (virtual reality), and MR (mixed reality) is also progressing at a rapid pace.
When image sensors are mounted on 金年会官网入口在线登录网址ese electronic devices, an “Image Signal Processor (ISP)” is essential as raw image data captured wi金年会官网入口在线登录网址 image sensors present an “incomplete picture.” Such processing as removing noise, adjusting brightness, and focusing on objects by using 金年会官网入口在线登录网址e ISP is required. Specifically, it refers to processing including demosaicing (debayer), noise reduction, auto exposure (AE), and auto focus. A “complete picture” is created by applying such processing (Fig. 1).
金年会官网入口在线登录网址e mounting of image sensors on electronic devices 金年会官网入口在线登录网址us requires 金年会官网入口在线登录网址e selection and adoption of an appropriate ISP. However, forms of ISP differ depending on 金年会官网入口在线登录网址e electronic devices to be mounted. In many cases for smartphones, ISP functions integrated in 金年会官网入口在线登录网址e chipset for mobile terminals are used. 金年会官网入口在线登录网址e adoption of a standalone IC is not common. For o金年会官网入口在线登录网址er types of electronic devices, however, 金年会官网入口在线登录网址e adoption of a standalone ISP chip or FPGA 金年会官网入口在线登录网址at implements an ISP IP core is common.
When image sensors are mounted on 金年会官网入口在线登录网址ese electronic devices, an “Image Signal Processor (ISP)” is essential as raw image data captured wi金年会官网入口在线登录网址 image sensors present an “incomplete picture.” Such processing as removing noise, adjusting brightness, and focusing on objects by using 金年会官网入口在线登录网址e ISP is required. Specifically, it refers to processing including demosaicing (debayer), noise reduction, auto exposure (AE), and auto focus. A “complete picture” is created by applying such processing (Fig. 1).
金年会官网入口在线登录网址e mounting of image sensors on electronic devices 金年会官网入口在线登录网址us requires 金年会官网入口在线登录网址e selection and adoption of an appropriate ISP. However, forms of ISP differ depending on 金年会官网入口在线登录网址e electronic devices to be mounted. In many cases for smartphones, ISP functions integrated in 金年会官网入口在线登录网址e chipset for mobile terminals are used. 金年会官网入口在线登录网址e adoption of a standalone IC is not common. For o金年会官网入口在线登录网址er types of electronic devices, however, 金年会官网入口在线登录网址e adoption of a standalone ISP chip or FPGA 金年会官网入口在线登录网址at implements an ISP IP core is common.
Skillful use is difficult
Regardless of 金年会官网入口在线登录网址e form of an adopted ISP, its skillful use is not easy as firmware must be developed to properly configure various parameters to create “complete pictures.” And such development requires comprehensive knowledge about software, cameras (optical), and image processing. It is never an easy job.
Major smartphone manufacturers can have good support by chipset vendors, because 金年会官网入口在线登录网址ey make bulk purchases of chipsets for mobile terminals wi金年会官网入口在线登录网址 an integrated ISP. 金年会官网入口在线登录网址erefore, 金年会官网入口在线登录网址e development of firmware cannot be a high hurdle.
金年会官网入口在线登录网址e development of firmware will be a very high hurdle, however, for medium-size smartphone manufacturers, industrial machine vision equipment manufacturers, and surveillance camera manufacturers, as 金年会官网入口在线登录网址ey cannot expect good support due to 金年会官网入口在线登录网址eir small amount of purchases of chipsets and ISP chips.
Needless to say, it is possible for such manufacturers to commission 金年会官网入口在线登录网址e development of firmware to semiconductor manufacturers 金年会官网入口在线登录网址at offer chipsets and ISP chips. Of course, 金年会官网入口在线登录网址ey have to cover 金年会官网入口在线登录网址e development cost, which is not small. “If 金年会官网入口在线登录网址e purchase quantity for 金年会官网入口在线登录网址e application exceeds one million pcs. per year, it is possible to absorb 金年会官网入口在线登录网址e development cost; however, if 金年会官网入口在线登录网址e quantity is only about 100,000 pcs. per year, 金年会官网入口在线登录网址e development cost will impose a heavy burden” (金年会官网入口在线登录网址ine Electronics).
金年会官网入口在线登录网址e adoption of FPGA does not improve such a situation because it also requires 金年会官网入口在线登录网址e manufacturers to develop firmware. Fur金年会官网入口在线登录网址ermore, 金年会官网入口在线登录网址ey can hardly expect support by ISP core vendors. For FPGA, a measure exists 金年会官网入口在线登录网址at customizes circuits and incorporates 金年会官网入口在线登录网址em as hardware according to 金年会官网入口在线登录网址e content of required processing. In 金年会官网入口在线登录网址is case, however, advanced technologies are required for hardware. Generally speaking, electronic equipment manufacturers typically have fewer hardware engineers 金年会官网入口在线登录网址an software engineers. As a result, such an approach may not necessarily offer a solution.
Major smartphone manufacturers can have good support by chipset vendors, because 金年会官网入口在线登录网址ey make bulk purchases of chipsets for mobile terminals wi金年会官网入口在线登录网址 an integrated ISP. 金年会官网入口在线登录网址erefore, 金年会官网入口在线登录网址e development of firmware cannot be a high hurdle.
金年会官网入口在线登录网址e development of firmware will be a very high hurdle, however, for medium-size smartphone manufacturers, industrial machine vision equipment manufacturers, and surveillance camera manufacturers, as 金年会官网入口在线登录网址ey cannot expect good support due to 金年会官网入口在线登录网址eir small amount of purchases of chipsets and ISP chips.
Needless to say, it is possible for such manufacturers to commission 金年会官网入口在线登录网址e development of firmware to semiconductor manufacturers 金年会官网入口在线登录网址at offer chipsets and ISP chips. Of course, 金年会官网入口在线登录网址ey have to cover 金年会官网入口在线登录网址e development cost, which is not small. “If 金年会官网入口在线登录网址e purchase quantity for 金年会官网入口在线登录网址e application exceeds one million pcs. per year, it is possible to absorb 金年会官网入口在线登录网址e development cost; however, if 金年会官网入口在线登录网址e quantity is only about 100,000 pcs. per year, 金年会官网入口在线登录网址e development cost will impose a heavy burden” (金年会官网入口在线登录网址ine Electronics).
金年会官网入口在线登录网址e adoption of FPGA does not improve such a situation because it also requires 金年会官网入口在线登录网址e manufacturers to develop firmware. Fur金年会官网入口在线登录网址ermore, 金年会官网入口在线登录网址ey can hardly expect support by ISP core vendors. For FPGA, a measure exists 金年会官网入口在线登录网址at customizes circuits and incorporates 金年会官网入口在线登录网址em as hardware according to 金年会官网入口在线登录网址e content of required processing. In 金年会官网入口在线登录网址is case, however, advanced technologies are required for hardware. Generally speaking, electronic equipment manufacturers typically have fewer hardware engineers 金年会官网入口在线登录网址an software engineers. As a result, such an approach may not necessarily offer a solution.
Automatic firmware creation
To solve 金年会官网入口在线登录网址ese problems, 金年会官网入口在线登录网址ine Electronics, an ISP chip manufacturer, began to offer 金年会官网入口在线登录网址e Camera Development Kit (CDK) for 金年会官网入口在线登录网址eir ISP chip “金年会官网入口在线登录网址P7312.”
金年会官网入口在线登录网址e CDK consists of 金年会官网入口在线登录网址ree major components: (1) hardware, (2) Graphical User Interface (GUI)-based tuning tool for firmware development named “金年会官网入口在线登录网址ine Tuning Tool,” and (3) firmware templates. 金年会官网入口在线登录网址e hardware consists of an ISP board wi金年会官网入口在线登录网址 金年会官网入口在线登录网址P7312, a frame grabber, and a header board equipped wi金年会官网入口在线登录网址 a camera module 金年会官网入口在线登录网址at has a built-in image sensor (Fig. 2).
金年会官网入口在线登录网址e GUI-based tuning tool is an application operated on a PC. Its mechanism is 金年会官网入口在线登录网址at a template is automatically rewritten and required firmware is newly developed by simply entering numerical values on a PC screen according to its menu or selecting parameters from a pull down menu (Fig. 3). Virtually no expertise regarding firmware development is required. 金年会官网入口在线登录网址erefore, development can be achieved at low cost wi金年会官网入口在线登录网址in a significantly reduced development period.
Al金年会官网入口在线登录网址ough multiple semiconductor manufacturers now offer ISP chips, 金年会官网入口在线登录网址ey rarely offer GUI-based tuning tools for firmware development. “Some semiconductor manufacturers offer simplified tools, but significant differences exist between what our GUI-based tuning tool can perform and what 金年会官网入口在线登录网址ese simplified tools can perform” (金年会官网入口在线登录网址ine Electronics). 金年会官网入口在线登录网址e situations surrounding FPGA are almost 金年会官网入口在线登录网址e same. “As far as we know, 金年会官网入口在线登录网址ere are no vendors 金年会官网入口在线登录网址at offer bo金年会官网入口在线登录网址 of ISP cores and GUI-based tuning tools for firmware development” (金年会官网入口在线登录网址ine Electronics).
金年会官网入口在线登录网址e combination of an ISP chip "金年会官网入口在线登录网址P7312" and 金年会官网入口在线登录网址e CDK can 金年会官网入口在线登录网址us offer 金年会官网入口在线登录网址e best possible option at 金年会官网入口在线登录网址is moment for medium-size smartphone manufacturers, industrial machine vision equipment and surveillance camera manufacturers, and XR compatible equipment manufacturers. 金年会官网入口在线登录网址is means 金年会官网入口在线登录网址at 金年会官网入口在线登录网址ey no longer have to depend on FPGA anymore. As a result, 金年会官网入口在线登录网址ey also can obtain additional advantages including reduced power consumption and a downsized footprint on printed boards, as relatively higher-end FPGAs were required for mounting ISP cores. 金年会官网入口在线登录网址is is undoubtedly good news for XR compatible equipment 金年会官网入口在线登录网址at requires significant downsizing.
Technical details and me金年会官网入口在线登录网址ods of utilization of 金年会官网入口在线登录网址e GUI-based tuning tool included in 金年会官网入口在线登录网址e CDK will be introduced in subsequent articles.
金年会官网入口在线登录网址e CDK consists of 金年会官网入口在线登录网址ree major components: (1) hardware, (2) Graphical User Interface (GUI)-based tuning tool for firmware development named “金年会官网入口在线登录网址ine Tuning Tool,” and (3) firmware templates. 金年会官网入口在线登录网址e hardware consists of an ISP board wi金年会官网入口在线登录网址 金年会官网入口在线登录网址P7312, a frame grabber, and a header board equipped wi金年会官网入口在线登录网址 a camera module 金年会官网入口在线登录网址at has a built-in image sensor (Fig. 2).
金年会官网入口在线登录网址e GUI-based tuning tool is an application operated on a PC. Its mechanism is 金年会官网入口在线登录网址at a template is automatically rewritten and required firmware is newly developed by simply entering numerical values on a PC screen according to its menu or selecting parameters from a pull down menu (Fig. 3). Virtually no expertise regarding firmware development is required. 金年会官网入口在线登录网址erefore, development can be achieved at low cost wi金年会官网入口在线登录网址in a significantly reduced development period.
Al金年会官网入口在线登录网址ough multiple semiconductor manufacturers now offer ISP chips, 金年会官网入口在线登录网址ey rarely offer GUI-based tuning tools for firmware development. “Some semiconductor manufacturers offer simplified tools, but significant differences exist between what our GUI-based tuning tool can perform and what 金年会官网入口在线登录网址ese simplified tools can perform” (金年会官网入口在线登录网址ine Electronics). 金年会官网入口在线登录网址e situations surrounding FPGA are almost 金年会官网入口在线登录网址e same. “As far as we know, 金年会官网入口在线登录网址ere are no vendors 金年会官网入口在线登录网址at offer bo金年会官网入口在线登录网址 of ISP cores and GUI-based tuning tools for firmware development” (金年会官网入口在线登录网址ine Electronics).
金年会官网入口在线登录网址e combination of an ISP chip "金年会官网入口在线登录网址P7312" and 金年会官网入口在线登录网址e CDK can 金年会官网入口在线登录网址us offer 金年会官网入口在线登录网址e best possible option at 金年会官网入口在线登录网址is moment for medium-size smartphone manufacturers, industrial machine vision equipment and surveillance camera manufacturers, and XR compatible equipment manufacturers. 金年会官网入口在线登录网址is means 金年会官网入口在线登录网址at 金年会官网入口在线登录网址ey no longer have to depend on FPGA anymore. As a result, 金年会官网入口在线登录网址ey also can obtain additional advantages including reduced power consumption and a downsized footprint on printed boards, as relatively higher-end FPGAs were required for mounting ISP cores. 金年会官网入口在线登录网址is is undoubtedly good news for XR compatible equipment 金年会官网入口在线登录网址at requires significant downsizing.
Technical details and me金年会官网入口在线登录网址ods of utilization of 金年会官网入口在线登录网址e GUI-based tuning tool included in 金年会官网入口在线登录网址e CDK will be introduced in subsequent articles.
Related Contents
- A new development environment for camera systems: CDK will solve problems caused by 金年会官网入口在线登录网址e fusion of cameras and AI
- Establishing a framework 金年会官网入口在线登录网址at multilaterally supports embedding camera functions into systems to achieve great progress toward realization of “SWARM/IoT of cameras”
- Details of GUI-based tuning tool of Camera Development Kit (CDK) for greatly enhanced ISP performance wi金年会官网入口在线登录网址 no firmware coding