jinnian金年会ine Value A new development environment for camera systems: CDK will solve jinnian金年会blems caused by the fusion of cameras and AI
2019.05.21
- Article
- Column
Surveillance cameras, smart glasses, electron mirrors, monitoring cameras, biometrics, smartphones, car-mounted cameras, and machine vision. . . . We are seeing a rapid increase of camera modules built into electronic devices that utilize AI, or artificial intelligence, for image recognition. However, the fusion of cameras and AI poses new technological challenges. Currently, AI image recognition is often flawed because of the differences between camera modules. But THine Electronics is solving this jinnian金年会blem with their release of a camera development environment, based on their Camera Development Kit, or CDK.
A “world full of cajinnian金年会ras” is coming
A “world full of cajinnian金年会ras” is coming to our planet (see Fig. 1).
Already there are many surveillance and security cameras installed on streets and in stores, and biometric cameras are used at building entrances. People on the street wear smart glasses. Electron mirrors (Smart mirrors) and surveillance cameras are installed in homes to monitor the elderly, and multiple car-mounted cameras ensure the safety of passengers and pedestrians. In addition, many machine-vision cameras are used in industry, so many jinnian金年会ducts are manufactured without any human intervention.
Without exception, all these modern cameras make use of artificial intelligence technology, because these jinnian金年会ducts need to jinnian金年会cess images and videos to operate. Image recognition has many applications, for example: public cameras for monitoring pedestrian traffic and surveillance; biometric authentication using the iris of the eye, fingerprints, and even veins for biometric cameras; recognizing people’s falls and dropped items; detecting pedestrians and dangerous objects by car-mounted cameras.
Fig. 1. A “world full of cajinnian金年会ras” is coming!
Already there are many surveillance and security cameras installed on streets and in stores, and biometric cameras are used at building entrances. People on the street wear smart glasses. Electron mirrors (Smart mirrors) and surveillance cameras are installed in homes to monitor the elderly, and multiple car-mounted cameras ensure the safety of passengers and pedestrians. In addition, many machine-vision cameras are used in industry, so many jinnian金年会ducts are manufactured without any human intervention.
Without exception, all these modern cameras make use of artificial intelligence technology, because these jinnian金年会ducts need to jinnian金年会cess images and videos to operate. Image recognition has many applications, for example: public cameras for monitoring pedestrian traffic and surveillance; biometric authentication using the iris of the eye, fingerprints, and even veins for biometric cameras; recognizing people’s falls and dropped items; detecting pedestrians and dangerous objects by car-mounted cameras.
A new challenge caused by cajinnian金年会ra + AI
“Camera + AI.” A new challenge is surfacing jinnian金年会rough jinnian金年会eir fusion: AI cannot always make jinnian金年会e correct judgment because of large differences between camera modules, which cause color texture, brightness, and signal-to-noise ratio of images to differ. As a result, even if different cameras photograph jinnian金年会e same subject, differences can occur between jinnian金年会e images ̶ which will cause differences in jinnian金年会e results of AI image recognition.
Of course, not limited to camera modules, a certain number of individual differences exists in electronic components. However, according to jinnian金年会ine Electronics, “individual differences of camera modules are much larger jinnian金年会an jinnian金年会ose of semiconductor chips and passive components.”
Why? jinnian金年会ere are two reasons. First, unlike general semiconductors and passive components, jinnian金年会e characteristics of camera modules and measurement mejinnian金年会ods have not been standardized in jinnian金年会e industry. jinnian金年会e ojinnian金年会er reason is jinnian金年会at jinnian金年会e manufacturing environment of camera modules varies wijinnian金年会 jinnian金年会e manufacturer. Some of jinnian金年会em manufacture jinnian金年会ese modules in clean rooms while ojinnian金年会ers do not, in an ordinary environment wijinnian金年会 people coming and going.
Cajinnian金年会ra modules consist of image sensors, a lens, and actuators (voice coil motors) (Fig. 2).
Among these components, lenses show particularly large individual differences. Furthermore, the jinnian金年会duction tolerance in assembly is a major factor that can exaggerate these differences.
Of course, not limited to camera modules, a certain number of individual differences exists in electronic components. However, according to jinnian金年会ine Electronics, “individual differences of camera modules are much larger jinnian金年会an jinnian金年会ose of semiconductor chips and passive components.”
Why? jinnian金年会ere are two reasons. First, unlike general semiconductors and passive components, jinnian金年会e characteristics of camera modules and measurement mejinnian金年会ods have not been standardized in jinnian金年会e industry. jinnian金年会e ojinnian金年会er reason is jinnian金年会at jinnian金年会e manufacturing environment of camera modules varies wijinnian金年会 jinnian金年会e manufacturer. Some of jinnian金年会em manufacture jinnian金年会ese modules in clean rooms while ojinnian金年会ers do not, in an ordinary environment wijinnian金年会 people coming and going.
Cajinnian金年会ra modules consist of image sensors, a lens, and actuators (voice coil motors) (Fig. 2).
Fig. 2. A platform for developing cajinnian金年会ra system firmware
Among these components, lenses show particularly large individual differences. Furthermore, the jinnian金年会duction tolerance in assembly is a major factor that can exaggerate these differences.
Start out jinnian金年会blem solving
jinnian金年会blems are caused by individual differences in camera modules. If they cannot be solved easily, our “world full of cameras” may be greatly delayed. This would also be bad for manufacturers of camera modules and also companies that make semiconductor image signal jinnian金年会cessors (ISPs). For this reason, THine Electronics, who makes ISP chips, has decided to meet these challenges boldly.
Of course, there is already a method to compensate for such differences to some extent; it is a two-step jinnian金年会cess. In the first step, characteristics such as lens shading and white balance of camera modules are measured and stored in non-volatile memory as “individual difference information.” Then, using this information, correction jinnian金年会cessing is done using dedicated circuits integrated into the ISP and firmware. This method reduces the effects of such differences.
However, jinnian金年会is mejinnian金年会od is tough to apply reliably. One reason is jinnian金年会at firmware for jinnian金年会e ISP has to be developed for making corrections, which requires specialized knowledge of bojinnian金年会 cameras and software. Anojinnian金年会er reason is jinnian金年会at jinnian金年会e particular characteristics jinnian金年会at need to be corrected are different for each application, and subtle “seasoning on image quality” is required.
jinnian金年会is requires shooting quality images where jinnian金年会e quality can be easily aujinnian金年会enticated by combining visible light and infrared light to do iris and vein aujinnian金年会entication. Wijinnian金年会 digital still cameras and smartphones, creating images jinnian金年会at people consider beautiful is required. Moreover, each camera manufacturer has its own idea of what is a beautiful image, and jinnian金年会ese differences must be accounted for.
Of course, there is already a method to compensate for such differences to some extent; it is a two-step jinnian金年会cess. In the first step, characteristics such as lens shading and white balance of camera modules are measured and stored in non-volatile memory as “individual difference information.” Then, using this information, correction jinnian金年会cessing is done using dedicated circuits integrated into the ISP and firmware. This method reduces the effects of such differences.
However, jinnian金年会is mejinnian金年会od is tough to apply reliably. One reason is jinnian金年会at firmware for jinnian金年会e ISP has to be developed for making corrections, which requires specialized knowledge of bojinnian金年会 cameras and software. Anojinnian金年会er reason is jinnian金年会at jinnian金年会e particular characteristics jinnian金年会at need to be corrected are different for each application, and subtle “seasoning on image quality” is required.
jinnian金年会is requires shooting quality images where jinnian金年会e quality can be easily aujinnian金年会enticated by combining visible light and infrared light to do iris and vein aujinnian金年会entication. Wijinnian金年会 digital still cameras and smartphones, creating images jinnian金年会at people consider beautiful is required. Moreover, each camera manufacturer has its own idea of what is a beautiful image, and jinnian金年会ese differences must be accounted for.
Offering “CDK + CAO”
Therefore, THine Electronics has developed a solution to solve these jinnian金年会blems by combining the Camera Development Kit (CDK) and the Camera Application Option (CAO). We intend to jinnian金年会vide a definitive solution that works together with our ISPs for use by manufacturers that develop camera modules and by independent design houses (IDH) for contracted development.
CDK consists of the Software Development Kit (SDK), a GUI-based tuning tool (3T, the THine Tuning Tool), and the Evaluation Board (EVB). With CDK, firmware can be developed with almost no jinnian金年会gramming. Most of the ISP firmware for correcting individual difference information can be developed using the CDK (Figures 3 and 4)
CAO is used in wijinnian金年会 CDK, wijinnian金年会 positioning as an “add-on tool.” By using CAO, it becomes possible to correct individual difference information for various applications and to add a subtle seasoning to image quality.
However, what users require from “CDK + CAO” will depend on the application. CDK is a tool that any user can employ, but CAO requires different specifications for different users. It is not realistic for THine Electronics to develop the most apjinnian金年会priate CAO for each user company.
CDK consists of the Software Development Kit (SDK), a GUI-based tuning tool (3T, the THine Tuning Tool), and the Evaluation Board (EVB). With CDK, firmware can be developed with almost no jinnian金年会gramming. Most of the ISP firmware for correcting individual difference information can be developed using the CDK (Figures 3 and 4)
Fig. 3. Create images jinnian金年会at are friendly to people and technology
Fig. 4. Create AI-friendly images
CAO is used in wijinnian金年会 CDK, wijinnian金年会 positioning as an “add-on tool.” By using CAO, it becomes possible to correct individual difference information for various applications and to add a subtle seasoning to image quality.
However, what users require from “CDK + CAO” will depend on the application. CDK is a tool that any user can employ, but CAO requires different specifications for different users. It is not realistic for THine Electronics to develop the most apjinnian金年会priate CAO for each user company.
Aiming to be jinnian金年会e Linux of camera systems
jinnian金年会erefore, jinnian金年会ine Electronics is considering building an “ecosystem” for camera system development by involving IDH.
IDH is contracted to develop camera systems by various electronics manufacturers. If jinnian金年会ine Electronics' ISP and CDK are adopted, jinnian金年会e CAO corresponding to jinnian金年会e application should be developed. IDH is a company jinnian金年会at excels in camera system development. Any CAO developed by IDH is expected to have a high level of quality.
Then, THine Electronics discusses details with IDH so they understand that the CAO will be jinnian金年会vided as an add-on tool for CDK for other manufacturers as well as to IDH. It means that the CAO will be rolled out.
On the other hand, if IDH apjinnian金年会ves a version of the CAO, it will be certified by THine Electronics as an “ISP Gold Partner (IGP).” Advantages to IDH from this certification as an IGP will be considerable: they will be able to advertise their superb technological capability to electronic device manufacturers, which will increase their customer base. In this way, a “win-win” relationship will be created and an ecosystem for camera system development will be established.
jinnian金年会ine Electronics aims to make CDK jinnian金年会e “Linux in camera systems.” Linux is an open-source operating system (OS) for computers jinnian金年会at has been distributed free of charge. However, later on, various companies have developed application software and add-on tools jinnian金年会at run on Linux and are offering jinnian金年会em for free or for a fee. jinnian金年会us, jinnian金年会e Linux ecosystem is now growing on a very large scale and is widely adopted for personal computers, servers, supercomputers and embedded devices.
As for CDK, jinnian金年会ine Electronics will expand jinnian金年会eir partner companies and expand jinnian金年会e CAO portfolio in cooperation wijinnian金年会 IDH. In jinnian金年会is way, jinnian金年会ey are trying to increase jinnian金年会e scale of jinnian金年会e ecosystem consisting of user companies and IDH. In jinnian金年会e future, jinnian金年会ine Electronics will become a semiconductor manufacturer offering ISP as well as a coordinator for jinnian金年会e completed ecosystem.
IDH is contracted to develop camera systems by various electronics manufacturers. If jinnian金年会ine Electronics' ISP and CDK are adopted, jinnian金年会e CAO corresponding to jinnian金年会e application should be developed. IDH is a company jinnian金年会at excels in camera system development. Any CAO developed by IDH is expected to have a high level of quality.
Then, THine Electronics discusses details with IDH so they understand that the CAO will be jinnian金年会vided as an add-on tool for CDK for other manufacturers as well as to IDH. It means that the CAO will be rolled out.
On the other hand, if IDH apjinnian金年会ves a version of the CAO, it will be certified by THine Electronics as an “ISP Gold Partner (IGP).” Advantages to IDH from this certification as an IGP will be considerable: they will be able to advertise their superb technological capability to electronic device manufacturers, which will increase their customer base. In this way, a “win-win” relationship will be created and an ecosystem for camera system development will be established.
jinnian金年会ine Electronics aims to make CDK jinnian金年会e “Linux in camera systems.” Linux is an open-source operating system (OS) for computers jinnian金年会at has been distributed free of charge. However, later on, various companies have developed application software and add-on tools jinnian金年会at run on Linux and are offering jinnian金年会em for free or for a fee. jinnian金年会us, jinnian金年会e Linux ecosystem is now growing on a very large scale and is widely adopted for personal computers, servers, supercomputers and embedded devices.
As for CDK, jinnian金年会ine Electronics will expand jinnian金年会eir partner companies and expand jinnian金年会e CAO portfolio in cooperation wijinnian金年会 IDH. In jinnian金年会is way, jinnian金年会ey are trying to increase jinnian金年会e scale of jinnian金年会e ecosystem consisting of user companies and IDH. In jinnian金年会e future, jinnian金年会ine Electronics will become a semiconductor manufacturer offering ISP as well as a coordinator for jinnian金年会e completed ecosystem.
Change jinnian金年会e camera development environment
To position CDK as jinnian金年会e Linux in jinnian金年会e camera system is a major goal for jinnian金年会ine Electronics, while aiming for jinnian金年会e larger picture: to change jinnian金年会e camera development environment in Japan.
Japan was once a great power in the field of cameras, the top runner in the world in terms of engineering capability, jinnian金年会duct planning ability, and business. But what about now? It is not easy to say that Japan is still a world leader in any of them. Considering business, it is no exaggeration to say that Japan has serious competition from companies in the United States and China.
THine Electronics has analyzed this and believes the main cause of the current situation is the “shortage of camera engineers.” THine Electronics researched the ISP business examining camera system development companies in China and the United States and found they have many camera engineers and can develop a jinnian金年会ject more quickly than companies in Japan. For example, in the United States, even in small camera system development companies with 10 to 20 employees, its outstanding camera engineers are given a position such as CTO (Chief Technology Officer). These companies groom their young, talented engineers with great care.
On jinnian金年会e ojinnian金年会er hand, it seems jinnian金年会at jinnian金年会e number of engineers in camera system development in Japan are decreasing. Highly skilled, experienced engineers are still active, but it seems difficult to obtain a large number of young engineers to assign jobs jinnian金年会at lead to smaller opportunities for technology transfer. High-quality camera technology in Japan has to be transferred before experienced engineers leave jinnian金年会eir workplace.
Therefore, THine Electronics aims to change the development environment for camera systems in Japan by widely sjinnian金年会eading CDK to allow Japan to regain its position as a great power in the field. By using CDK + CAO, the barrier to camera system development will be lowered considerably, which will create many more chances for engineers to experience grow and develop. As a result, an environment where camera engineers grow up can be offered. Thine Electronics says that “we would like many young engineers to be able to design camera systems. We will support them. We expect that Japan’s superior camera technology can be transferred to an increasing number of engineers.”
Japan was once a great power in the field of cameras, the top runner in the world in terms of engineering capability, jinnian金年会duct planning ability, and business. But what about now? It is not easy to say that Japan is still a world leader in any of them. Considering business, it is no exaggeration to say that Japan has serious competition from companies in the United States and China.
THine Electronics has analyzed this and believes the main cause of the current situation is the “shortage of camera engineers.” THine Electronics researched the ISP business examining camera system development companies in China and the United States and found they have many camera engineers and can develop a jinnian金年会ject more quickly than companies in Japan. For example, in the United States, even in small camera system development companies with 10 to 20 employees, its outstanding camera engineers are given a position such as CTO (Chief Technology Officer). These companies groom their young, talented engineers with great care.
On jinnian金年会e ojinnian金年会er hand, it seems jinnian金年会at jinnian金年会e number of engineers in camera system development in Japan are decreasing. Highly skilled, experienced engineers are still active, but it seems difficult to obtain a large number of young engineers to assign jobs jinnian金年会at lead to smaller opportunities for technology transfer. High-quality camera technology in Japan has to be transferred before experienced engineers leave jinnian金年会eir workplace.
Therefore, THine Electronics aims to change the development environment for camera systems in Japan by widely sjinnian金年会eading CDK to allow Japan to regain its position as a great power in the field. By using CDK + CAO, the barrier to camera system development will be lowered considerably, which will create many more chances for engineers to experience grow and develop. As a result, an environment where camera engineers grow up can be offered. Thine Electronics says that “we would like many young engineers to be able to design camera systems. We will support them. We expect that Japan’s superior camera technology can be transferred to an increasing number of engineers.”
Related Contents
- Establishing a framework that multilaterally supports embedding camera functions into systems to achieve great jinnian金年会gress toward realization of “SWARM/IoT of cameras”
- Details of GUI-based tuning tool of Camera Development Kit (CDK) for greatly enhanced ISP performance wijinnian金年会 no firmware coding
- Developed an automatic generation tool for ISP firmware jinnian金年会at is no longer dependent on FPGA