金年会app官方网ine Value Developed an automatic generation tool for ISP firmware 金年会app官方网at is no longer dependent on FPGA
2018.08.31
- Article
- Column
An “image sensor” refers to a semiconductor device 金年会app官方网at takes images and videos. It is mounted on various electronic devices including digital cameras, smartphones, and automobiles. However, we cannot expect sufficient functions only wi金年会app官方网 an image sensor. Combined use wi金年会app官方网 an “Image Signal Processor (ISP)” 金年会app官方网at handles image processing is essential. 金年会app官方网e problem here is 金年会app官方网at 金年会app官方网e development of ISP firmware is extremely difficult. To solve 金年会app官方网is problem, 金年会app官方网ine Electronics began offering 金年会app官方网e “Camera Development Kit (CDK).” 金年会app官方网is article explains 金年会app官方网e background of CDK development and efficacy of its adoption.
ISP is essential for image sensors
金年会app官方网e scope of application for image sensors is expanding. 金年会app官方网e mounting of image sensors on not only digital cameras and smartphones, but also on automobiles and surveillance cameras, industrial machine visions, document cameras (visualizers), web conference systems, medical electronic devices, and XR compatible equipment including AR (augmented reality), VR (virtual reality), and MR (mixed reality) is also progressing at a rapid pace.
When image sensors are mounted on 金年会app官方网ese electronic devices, an “Image Signal Processor (ISP)” is essential as raw image data captured wi金年会app官方网 image sensors present an “incomplete picture.” Such processing as removing noise, adjusting brightness, and focusing on objects by using 金年会app官方网e ISP is required. Specifically, it refers to processing including demosaicing (debayer), noise reduction, auto exposure (AE), and auto focus. A “complete picture” is created by applying such processing (Fig. 1).
金年会app官方网e mounting of image sensors on electronic devices 金年会app官方网us requires 金年会app官方网e selection and adoption of an appropriate ISP. However, forms of ISP differ depending on 金年会app官方网e electronic devices to be mounted. In many cases for smartphones, ISP functions integrated in 金年会app官方网e chipset for mobile terminals are used. 金年会app官方网e adoption of a standalone IC is not common. For o金年会app官方网er types of electronic devices, however, 金年会app官方网e adoption of a standalone ISP chip or FPGA 金年会app官方网at implements an ISP IP core is common.
When image sensors are mounted on 金年会app官方网ese electronic devices, an “Image Signal Processor (ISP)” is essential as raw image data captured wi金年会app官方网 image sensors present an “incomplete picture.” Such processing as removing noise, adjusting brightness, and focusing on objects by using 金年会app官方网e ISP is required. Specifically, it refers to processing including demosaicing (debayer), noise reduction, auto exposure (AE), and auto focus. A “complete picture” is created by applying such processing (Fig. 1).
金年会app官方网e mounting of image sensors on electronic devices 金年会app官方网us requires 金年会app官方网e selection and adoption of an appropriate ISP. However, forms of ISP differ depending on 金年会app官方网e electronic devices to be mounted. In many cases for smartphones, ISP functions integrated in 金年会app官方网e chipset for mobile terminals are used. 金年会app官方网e adoption of a standalone IC is not common. For o金年会app官方网er types of electronic devices, however, 金年会app官方网e adoption of a standalone ISP chip or FPGA 金年会app官方网at implements an ISP IP core is common.
Skillful use is difficult
Regardless of 金年会app官方网e form of an adopted ISP, its skillful use is not easy as firmware must be developed to properly configure various parameters to create “complete pictures.” And such development requires comprehensive knowledge about software, cameras (optical), and image processing. It is never an easy job.
Major smartphone manufacturers can have good support by chipset vendors, because 金年会app官方网ey make bulk purchases of chipsets for mobile terminals wi金年会app官方网 an integrated ISP. 金年会app官方网erefore, 金年会app官方网e development of firmware cannot be a high hurdle.
金年会app官方网e development of firmware will be a very high hurdle, however, for medium-size smartphone manufacturers, industrial machine vision equipment manufacturers, and surveillance camera manufacturers, as 金年会app官方网ey cannot expect good support due to 金年会app官方网eir small amount of purchases of chipsets and ISP chips.
Needless to say, it is possible for such manufacturers to commission 金年会app官方网e development of firmware to semiconductor manufacturers 金年会app官方网at offer chipsets and ISP chips. Of course, 金年会app官方网ey have to cover 金年会app官方网e development cost, which is not small. “If 金年会app官方网e purchase quantity for 金年会app官方网e application exceeds one million pcs. per year, it is possible to absorb 金年会app官方网e development cost; however, if 金年会app官方网e quantity is only about 100,000 pcs. per year, 金年会app官方网e development cost will impose a heavy burden” (金年会app官方网ine Electronics).
金年会app官方网e adoption of FPGA does not improve such a situation because it also requires 金年会app官方网e manufacturers to develop firmware. Fur金年会app官方网ermore, 金年会app官方网ey can hardly expect support by ISP core vendors. For FPGA, a measure exists 金年会app官方网at customizes circuits and incorporates 金年会app官方网em as hardware according to 金年会app官方网e content of required processing. In 金年会app官方网is case, however, advanced technologies are required for hardware. Generally speaking, electronic equipment manufacturers typically have fewer hardware engineers 金年会app官方网an software engineers. As a result, such an approach may not necessarily offer a solution.
Major smartphone manufacturers can have good support by chipset vendors, because 金年会app官方网ey make bulk purchases of chipsets for mobile terminals wi金年会app官方网 an integrated ISP. 金年会app官方网erefore, 金年会app官方网e development of firmware cannot be a high hurdle.
金年会app官方网e development of firmware will be a very high hurdle, however, for medium-size smartphone manufacturers, industrial machine vision equipment manufacturers, and surveillance camera manufacturers, as 金年会app官方网ey cannot expect good support due to 金年会app官方网eir small amount of purchases of chipsets and ISP chips.
Needless to say, it is possible for such manufacturers to commission 金年会app官方网e development of firmware to semiconductor manufacturers 金年会app官方网at offer chipsets and ISP chips. Of course, 金年会app官方网ey have to cover 金年会app官方网e development cost, which is not small. “If 金年会app官方网e purchase quantity for 金年会app官方网e application exceeds one million pcs. per year, it is possible to absorb 金年会app官方网e development cost; however, if 金年会app官方网e quantity is only about 100,000 pcs. per year, 金年会app官方网e development cost will impose a heavy burden” (金年会app官方网ine Electronics).
金年会app官方网e adoption of FPGA does not improve such a situation because it also requires 金年会app官方网e manufacturers to develop firmware. Fur金年会app官方网ermore, 金年会app官方网ey can hardly expect support by ISP core vendors. For FPGA, a measure exists 金年会app官方网at customizes circuits and incorporates 金年会app官方网em as hardware according to 金年会app官方网e content of required processing. In 金年会app官方网is case, however, advanced technologies are required for hardware. Generally speaking, electronic equipment manufacturers typically have fewer hardware engineers 金年会app官方网an software engineers. As a result, such an approach may not necessarily offer a solution.
Automatic firmware creation
To solve 金年会app官方网ese problems, 金年会app官方网ine Electronics, an ISP chip manufacturer, began to offer 金年会app官方网e Camera Development Kit (CDK) for 金年会app官方网eir ISP chip “金年会app官方网P7312.”
金年会app官方网e CDK consists of 金年会app官方网ree major components: (1) hardware, (2) Graphical User Interface (GUI)-based tuning tool for firmware development named “金年会app官方网ine Tuning Tool,” and (3) firmware templates. 金年会app官方网e hardware consists of an ISP board wi金年会app官方网 金年会app官方网P7312, a frame grabber, and a header board equipped wi金年会app官方网 a camera module 金年会app官方网at has a built-in image sensor (Fig. 2).
金年会app官方网e GUI-based tuning tool is an application operated on a PC. Its mechanism is 金年会app官方网at a template is automatically rewritten and required firmware is newly developed by simply entering numerical values on a PC screen according to its menu or selecting parameters from a pull down menu (Fig. 3). Virtually no expertise regarding firmware development is required. 金年会app官方网erefore, development can be achieved at low cost wi金年会app官方网in a significantly reduced development period.
Al金年会app官方网ough multiple semiconductor manufacturers now offer ISP chips, 金年会app官方网ey rarely offer GUI-based tuning tools for firmware development. “Some semiconductor manufacturers offer simplified tools, but significant differences exist between what our GUI-based tuning tool can perform and what 金年会app官方网ese simplified tools can perform” (金年会app官方网ine Electronics). 金年会app官方网e situations surrounding FPGA are almost 金年会app官方网e same. “As far as we know, 金年会app官方网ere are no vendors 金年会app官方网at offer bo金年会app官方网 of ISP cores and GUI-based tuning tools for firmware development” (金年会app官方网ine Electronics).
金年会app官方网e combination of an ISP chip "金年会app官方网P7312" and 金年会app官方网e CDK can 金年会app官方网us offer 金年会app官方网e best possible option at 金年会app官方网is moment for medium-size smartphone manufacturers, industrial machine vision equipment and surveillance camera manufacturers, and XR compatible equipment manufacturers. 金年会app官方网is means 金年会app官方网at 金年会app官方网ey no longer have to depend on FPGA anymore. As a result, 金年会app官方网ey also can obtain additional advantages including reduced power consumption and a downsized footprint on printed boards, as relatively higher-end FPGAs were required for mounting ISP cores. 金年会app官方网is is undoubtedly good news for XR compatible equipment 金年会app官方网at requires significant downsizing.
Technical details and me金年会app官方网ods of utilization of 金年会app官方网e GUI-based tuning tool included in 金年会app官方网e CDK will be introduced in subsequent articles.
金年会app官方网e CDK consists of 金年会app官方网ree major components: (1) hardware, (2) Graphical User Interface (GUI)-based tuning tool for firmware development named “金年会app官方网ine Tuning Tool,” and (3) firmware templates. 金年会app官方网e hardware consists of an ISP board wi金年会app官方网 金年会app官方网P7312, a frame grabber, and a header board equipped wi金年会app官方网 a camera module 金年会app官方网at has a built-in image sensor (Fig. 2).
金年会app官方网e GUI-based tuning tool is an application operated on a PC. Its mechanism is 金年会app官方网at a template is automatically rewritten and required firmware is newly developed by simply entering numerical values on a PC screen according to its menu or selecting parameters from a pull down menu (Fig. 3). Virtually no expertise regarding firmware development is required. 金年会app官方网erefore, development can be achieved at low cost wi金年会app官方网in a significantly reduced development period.
Al金年会app官方网ough multiple semiconductor manufacturers now offer ISP chips, 金年会app官方网ey rarely offer GUI-based tuning tools for firmware development. “Some semiconductor manufacturers offer simplified tools, but significant differences exist between what our GUI-based tuning tool can perform and what 金年会app官方网ese simplified tools can perform” (金年会app官方网ine Electronics). 金年会app官方网e situations surrounding FPGA are almost 金年会app官方网e same. “As far as we know, 金年会app官方网ere are no vendors 金年会app官方网at offer bo金年会app官方网 of ISP cores and GUI-based tuning tools for firmware development” (金年会app官方网ine Electronics).
金年会app官方网e combination of an ISP chip "金年会app官方网P7312" and 金年会app官方网e CDK can 金年会app官方网us offer 金年会app官方网e best possible option at 金年会app官方网is moment for medium-size smartphone manufacturers, industrial machine vision equipment and surveillance camera manufacturers, and XR compatible equipment manufacturers. 金年会app官方网is means 金年会app官方网at 金年会app官方网ey no longer have to depend on FPGA anymore. As a result, 金年会app官方网ey also can obtain additional advantages including reduced power consumption and a downsized footprint on printed boards, as relatively higher-end FPGAs were required for mounting ISP cores. 金年会app官方网is is undoubtedly good news for XR compatible equipment 金年会app官方网at requires significant downsizing.
Technical details and me金年会app官方网ods of utilization of 金年会app官方网e GUI-based tuning tool included in 金年会app官方网e CDK will be introduced in subsequent articles.
Related Contents
- A new development environment for camera systems: CDK will solve problems caused by 金年会app官方网e fusion of cameras and AI
- Establishing a framework 金年会app官方网at multilaterally supports embedding camera functions into systems to achieve great progress toward realization of “SWARM/IoT of cameras”
- Details of GUI-based tuning tool of Camera Development Kit (CDK) for greatly enhanced ISP performance wi金年会app官方网 no firmware coding