If you’ve ever driven a car, truck, motorcycle, or anything else on wheels in the city, you’ve gotten stuck in traffic. And it’s the worst. But new technology may have found a way to reduce the headache.

Researchers at Aston University in the U.K. have developed an artificial intelligence traffic light system that relies on live camera footage to make adjustments and keep traffic flowing. The system utilizes deep reinforcement learning, which means it understands when it's doing a lousy job and it varies its actions to gradually improve.

The researchers said this method improved upon the manually designed phase transitions that are often used for traffic lights.

To test the system, the researchers built a traffic simulator called Traffic 3D (which sounds like a horrible video game) to train the program and prepare it for handling different traffic and weather scenarios.

In reality, the traffic control system actually does function sort of like a game. Researchers said the program gets a "reward" when it gets a car through an intersection and if a car has to wait or there’s a traffic jam, there’s a negative reward. The researchers said they don’t supply any input, they "simply control the reward system," which can be changed to allow for emergency vehicles to pass through.

In subsequent real-world testing, the researchers said the system adapted well to real intersections. They hope to begin further testing the system on real roads this year.

The automotive industry is in the middle of a complex, expensive and revolutionary path to software-defined vehicles. Every company will need to develop, buy and manage a lot of software to remain leaders in this software-centric generation. This column will provide an overview of the factors and complexities that impact the journey to the software-defined vehicle era.

The first step is to understand that the auto industry has features that increase software complexity compared to most other industries.

The definition of “software-defined” means that a large majority of car functionality is now implemented by software applications that run on required processors, memory and sensors. Additionally, most of the functionality is defined by how well the human-machine interface is implemented in software.

There are numerous choices and questions when it comes to what paths to take regarding existing, emerging and new potential technologies. Additionally, regulations on software lifetime management have been introduced and more are expected; especially for autonomous driving software.

With all vehicles becoming connected, over-the-air software updates and cybersecurity software become a must. These technologies add complexity while also providing tremendous opportunities and advantages for auto manufacturers and vehicle users.

AUTO INDUSTRY SOFTWARE COMPLEXITIES

The auto industry has multiple characteristics that complicate the development, maintenance and management of its rapidly growing software portfolios. The next table summarizes these features with information on how they affect automotive software platforms, ultimately increasing the complexities of reaching a software-defined vehicle era.

Product Life

The lifetime of auto industry products is among the longest of any industry — at least, among volume products counted in the tens of millions of sales units per year. The auto software complexity and program size have increased dramatically and will continue to do so another decade or more. The sum of the numerous auto software platforms exceed 100 million lines of code in many vehicles today and may double or triple in the coming decade.

These factors will challenge OEMs and their suppliers in terms of software development, maintenance, bug fixes, recalls and updates across 10 to 15 years of customer use. This is complicated enough for one specific auto model that is usually updated every 3-4 years and this may happen from 2-4 times. The complexity increases tremendously when major OEMs have 10-20 models with some regional variations that are going through these model update cycles.

The transition from internal combustion engine vehicles (ICEVs) to battery electric vehicles (BEVs) adds another dimension to car software renewal and management. Developing new BEV models provides opportunities to start with a new, clean sheet of software instead of relying on legacy software platforms that may be antiquated and should be replaced with state-of-the-art software architectures.

Legacy software systems

The large amount of required legacy system expertise and maintenance demands for up to a decade is a challenge all auto OEMs and their suppliers face. Most of these legacy systems must be superseded by modern software platforms as new vehicle models are introduced.

This requires a lot of re-training and new expertise that will take time and money to acquire and develop. These constraints are significantly slowing down the switchover to software-defined vehicles for all OEMs and their suppliers.

Connected network of functions

For the past thirty years, auto electronics functions have grown into connected networks of functionality. Most new functions were an ECU with a microprocessor and relevant sensor as well as a growing amount of software over time. Many functions needed to communicate with other functions, and electronics buses were added. The networks are currently dominated by the CAN bus, but are expected to move to Ethernet-based networks at a growing rate.

As telematics and other connected car functions increased, the need for cybersecurity became a requirement and OTA software updates emerged as valuable functionality. By 2020, a high-end vehicle had over 50 ECUs connected via a multitude of interconnected buses. Further expanding this structure was not viable and the domain ECU became a better solution.

Domain ECU era

A domain ECU combines multiple small ECUs into a single ECU with a more powerful processor, larger memory and more capable software platforms and applications. Legacy systems are being replaced by domain ECUs and software-defined architectures. For some OEMs, this transition could take as long as a decade and most OEMs only started a few years ago.

The growth and capability of cloud-based software development platforms are speeding up creation of new software architectures and expanding their features and capabilities. The cloud-based approach is also adding software-as-a-service (SaaS) functionality at a rapid rate.

Real-time software

Many auto applications are classified as real-time software. This means that there are specific time limits to complete the software codes. Otherwise, the software that controls car operations, such as the engine, brakes, steering and acceleration, may fail and create safety issues. ADAS and AV functions are also examples of real-time software that is growing in importance.

The extra timing restrictions makes the development of real-time software more complex and costly compared to regular software.

Functional and AV safety

Functional safety is now a core feature for all real-time software platforms and is regulated by the ISO 26262 standard. Many software platforms must pass functional safety testing to be legally used in modern vehicles.

AV functions are next on this path, with new standards specifying how AV technology must be designed, in effect extending functional safety to automated driving systems. The key standards are ISO 21448, UL 4600 and IEEE P2851.

Software legislation

Key software legislation is focused on cybersecurity and OTA software update management. The UNECE WP.29 legislation passed in Europe in 2020 and regulates both cybersecurity and OTA software updates.

AI software

AI is growing in importance in the auto industry and will have a profound impact in the next decade. Specifically, AI tech advances are needed in the next decade and AI black box issues must be solved. The AV software driver depends on AI technology innovation. We are also counting on AI to improve software coding with fewer bugs, better efficiency and lower costs.

AV regulations

One of the more difficult issues to emerge is AV road legislation that will greatly impact future AV software. New laws, infrastructure and AV safety tracking systems are needed. These solutions often include difficult and controversial societal and political decisions on initial AV safety levels versus historical human driver safety. Multiple countries have started passing AV laws and much more is on the way.

Content usage

Content consumption in the car has grown dramatically in the last decade due to mobile device proliferation, with smartphones leading the parade. Auto OEMs attempted to develop their own software platforms to connect to smartphones but failed as software platforms from Apple and Google are now dominant.

Content usage rules vary between drivers and passengers due to driving distraction issues that are major causes of car crashes. AVs hold the promise of increased content consumption when available, which will expand the market opportunities for content software platforms in the auto industry.

AUTO OEM SOFTWARE PLATFORM PHASES

It is clear that the auto OEMs have a lot on their plates to become successful players in the software-defined vehicle era. The next figure is a simple block diagram summary of what OEMs must do over the next 15 years to become viable competitors, with two main stages: software developments in green blocks and the customers’ use phase of the software platforms in red blocks. The development phase for most software platforms takes 1-3 years, while the software platform usage stage is much longer at 10-15 years.

The auto industry is already levering software development platforms originally created for IT and other industries. The Integrated Development Environment platforms such as Eclipse are heavily used to create auto software platforms. Cloud platforms focused on software development have also grown strongly in recent years, with AWS and Microsoft Azure as the leaders.

A new approach to developing software known as “no-code” or “low-code” has emerged in the IT industry. It is based on creating higher-level developmental platforms that simplifies the process of creating software code. At the top of this trend is AI-based code generation. This trend is expected to see growing impact on automotive software development.

Another approach is to tailor software development to a specific application segment. Apex.AI, for example, focuses on functional safety software platforms.

All these software development platforms are used to create a large variety of automotive software platforms as shown in the above figure. Each vehicle family will have a portfolio of software platforms as shown in the two green boxes labelled Vehicle Family #1 and #N. This implies there are several additional vehicle families.

The more software platforms that can be shared and re-used across models and generations, the better the economics of the auto software business model becomes. In the past, this was a low priority on many auto OEMs’ strategy list. Now it is required, and all the auto OEMs are focusing on leveraging software platforms as much as they can.

In the above figure, the red blocks show the software platform use phase by vehicle customers with similar labels of Vehicle Family #1 and #N. The top red block shows the cloud-based platforms required to manage operations, updates and other activities the OEMs will need for a profitable software business.

The figure also lists the typical sales range of vehicle family platforms with high volume in the range of 0.5-2 million units per year. Low volume platforms are in the 50K to 150K yearly sales range.

There are many more details needed that go beyond the scope of this short article. Future perspectives and analysis may be worthwhile topics.

FINAL THOUGHTS

The auto industry is on a path to provide software-defined vehicles that will greatly improve functionalities that will continue to expand during their lifetimes. To get there is a challenge that the OEMs and suppliers are attacking with expanded technologies and new business models. Advancing software development platforms with a better fit to automotive complexities are especially important and are starting to appear.

A combination of buying some software platforms and insourcing other software platforms seems to be a common strategy. Growing use of cloud-based software development platforms is a favorite approach across the board.

Many questions remain on which paths are winning strategies and which OEMs and auto suppliers will remain leaders in the new software-defined vehicle era. How important will the high-tech industry become as their core software competencies become the driving forces of the automotive and transportation industries?

It’s becoming more common for manufacturers to use robots in quality control. Let’s explore the specific benefits these high-tech machines bring.

Making measurements more manageable

Measurement checks are critical parts of many quality control processes. However, they can become more complex when it involves handling delicate components. That’s why some people are interested in letting robots take over instead.

At OptiPro, a precision optical manufacturing and metrology equipment manufacturer, leaders sought to build a solution that would let customers automate the measurement of optical lenses. The previous method required users to manually measure the components before feeding them into a machine for final inspection. The automated alternative consisted of a collaborative robot fitted with a smart gripper.

A vital part of conducting a robot feasibility study involves exposing the machine to sample parts of various sizes, quality levels, and types. That way, the results are maximally useful in determining how the technology performs in the real world. The tests also verify that the robot won’t damage sensitive components.

“Part handling can be an issue for a lot of different brittle materials, ceramics and optics applications,” said Dave Mohring, metrology coordinator at OptiPro Systems. “Here, we’re implementing a robot that can automatically load and unload a part.”

First, the robotic gripper picks up and measures the component. It then sends the details to the coordinate measurement machine, which automatically loads the right software to check the piece.

After the coordinate measurement machine verifies the part’s specifics, it sends information to the robot based on whether it passed or failed the inspection. This improved process enables the checking of more components within a shorter time frame. It also frees up human operators to spend time on other tasks.

Robots take surface measurements with micron-scale precision

Another recent development involved researchers developing a lightweight optical system for the robotic 3D inspection of product surfaces. They believe their work could improve quality control processes for items like semiconductors and television components.

“Our system can measure 3D surface topographies with [an] unprecedented combination of flexibility, precision, and speed,” said project co-leader Daniel Wertjanz. “This creates less waste because manufacturing problems can be identified in real time and processes can be quickly adapted and optimized.”

The measurement system, which mounts to a robotic arm, is only about the size of an espresso cup. The team also accounted for the high-vibration environments of many manufacturing facilities, ensuring the technology would still work as expected in such conditions.

Contributing to sustainable reuse

People knowledgeable about electric-vehicle (EV) batteries caution that discussions must occur to ensure these power sources don’t end up discarded in landfills. They’re exploring various options for reuse and material recycling. Robots could play an important role in those activities by checking battery performance. Automated test results help people determine the health of those power sources and what to do with the retired batteries.

Researchers developed a robotic arm that helped conduct electrochemical impedance spectroscopy tests on a retired battery from a Nissan EV. They collected test data manually and then compared it with the results the robot recorded.

The outcomes showed that the machine got comparable results to the humans and needed no oversight during the process. The researchers said this achievement could assist in grading retired battery models without exposing people to potential dangers.

Letting robots keep quality levels high with battery disassembly

Robots can also support quality control by vastly improving the still largely inefficient process of battery disassembly. Researchers at the U.S. Department of Energy’s Oak Ridge National Laboratory developed a robotic system that protects people from the numerous hazards of this task, including toxic chemicals.

“Automatic disassembly of components containing critical materials not only eliminates labor-intensive manual disassembly but provides for an efficient process to separate the components into higher-value streams where the critical materials are concentrated into individual feedstocks for recycle processing,” said Tom Lograsso, director of the Department of Energy’s Critical Materials Institute. “This added value is an important part of establishing an economically viable process.”

People who worked on the project said significant bottlenecks exist when humans must manually disassemble battery packs because of the time involved. They said the robots are significantly faster. Additionally, humans typically bring more variability to a process than machines do. They get tired or may have insufficient training, which could impact the overall quality of the results. Robots can work continuously and maintain consistently reliable output.

Enabling continuous improvement

Robots have upended many conventional processes, helping people in various industries pursue higher quality while noticing other associated benefits. Robots can 3D-print structures in the construction sector, helping ensure they stay within strict specifications for a project.

Using robots in quality control is also valuable for determining whether a product will stand up to real-world use. Machines often reach that conclusion faster than humans because they subject an item to repetitive motions. Samsung relied on robots to test the durability of foldable phones. Doing so helped company employees see how parts failed and when. That information factored into decisions that improve the quality control of future devices.

The robot folded the phone approximately three times per second. After 119,380 folds, half the screen stopped working and a snap fell off the hinge. After 120,168 folds, the phone’s hinge got stuck and required extra force to open.

However, other testing conducted at Samsung showed the phone should last longer than that. The company’s representatives estimated that the device would tolerate 200,000 folds, making it last five years if people folded it 100 times per day. If it fails before then, as the robotic tests suggested, the actual device lifespan may be closer to three years.

Samsung’s engineers used those test results to scrutinize precisely what makes the phone fail faster than expected: Is it something to do with the external components, the inner workings, or a mix of both?

Additionally, using robots in quality control allows for getting the results more quickly. That allows team members to obtain the information to know whether certain design updates had the desired effects. This is a good example of how robots can support an organization’s culture of continuous improvement.

Relying on robots in quality control brings impressive results

These examples highlight why using robots in quality control processes is an increasingly desirable option. However, taking this approach is not a fast solution. It requires decision-makers to have a clear idea of what they want to achieve and be willing to devote the necessary resources to make it happen. Robots are not foolproof, but investing in high-tech options can often uncover challenges and improve the overall quality of critical or in-demand items.

AI chip startup Axelera has tested and validated a chip as a test vehicle for its Thetis digital in–memory compute core. The company’s tests show the 12–nm chip can achieve 39.3 TOPS with a power efficiency of 14.1 TOPS/W in an area of 9mm².

The test chip was taped out in December 2021, after an impressive four months’ design and verification, with the support of Imec IC–Link, a European application–specific integrated circuit solutions provider that is part of research institute Imec.

This test chip is a proof of concept for Axelera’s digital in–memory compute design, though the company has expertise in both digital and analog compute for AI acceleration (as well as RISC–V design), Axelera CEO Fabrizio Del Maffeo told EE Times.

Axelera also has access to an Imec analog computing technology which can achieve “thousands of TOPS/W,” Del Maffeo said, but networks need to be fine–tuned as small variances in analog components can affect the result.

“We have [analog computing] expertise, but we are also exploring interesting in–memory computing and other designs,” he said. “For our first product, the solution we’ve found is that since we want to target high precision and no retraining, it’s good to stay in the digital domain, but get the efficiency you typically get from being in the analog domain, with the same high throughput per area.”

The company’s tests revealed 39.3 TOPS of AI compute with an efficiency of 14.1 TOPS/W at INT8 precision when operating at 800 MHz. Throughput can be traded off energy efficiency via clock frequency; at the chip’s highest operating frequency, 970 MHz, the performance reached 48.16 TOPS. Peak energy efficiency — enhanced by taking advantage of highly sparse activations — reached 33 TOPS/W with the same test chip (operating at a different frequency).

Axelera’s test chip was one compute core; the company’s first product will be a multi–core design. The company previously said it would target “hundreds of TOPS” for its first product and this is still the plan.

“It won’t be a large number of cores, because we don’t need it,” Del Maffeo said. “The throughput we have and the efficiency we can reach… for edge applications, a few cores is enough.”

The company’s first markets will probably be the highly fragmented industrial, retail, and robotics AI industries, especially customers based close to home in Europe — this primarily includes medium–volume companies who don’t have extensive AI expertise in–house.

“TOPS and TOPS/W are great, but it’s not enough,” he said. “To win this market we have to deliver performance, usability, and price. In–memory compute can deliver on price because of its throughput per area… but that’s true only if you can solve the problem of usability.”

“For edge computing, 99.5% of customers have no clue what quantization is, and they don’t care,” he added. “They want to know if they can run their networks quickly on our chip — quickly means push a button and make it work.”

Axelera currently has 53 employees spread around Europe, including clusters in Eindhoven, The Netherlands; Leuven, Belgium; and Zurich, Switzerland. The company, incubated by BitFury since 2019, currently has 53 employees and expects to expand to around 65 this summer.

Axelera plans to announce its first product in the autumn ready for delivery to early–access customers in early 2023.

Chairman Kim Chang-ho is a man of many hats: one as Chairman of the Global Robot Cluster (GRC), another as Chairman of the Robot Enterprise Promotion Association (REPA), three more as Founder, Director and CEO of AJINEXTEK Co., Ltd., and yet another, maybe the one he’s proudest of, as hometown hero successfully promoting Daegu City since 2010 into a robotics hub, and now, after an intense national competition, as the mega-site for Korea’s massive Robot Technopolis.

With the official announcement in August, Daegu City as Robotics Technopolis serves as a brilliant capstone to Chairman Kim’s ten years of persistent work building out the city’s robotics ecosystem.

With over 100 robotics companies, educational institutions and R&D facilities already resident in Daegu City, and with another 500 projected between 2022 and 2028, the Daegu Robot Technopolis, known officially as the National Robot Innovation Project, might well become the largest robot city in Asia.

The rise of Daegu as Robot Technopolis begins in earnest this November 17 with the annual board meeting and general assembly of the Global Robot Cluster (GRC), founded by Chairman Kim, and built out to now include twenty members from seventeen countries.

The GRC and its member delegations, together with Daegu’s Mayor Kwon Young-jin, who was instrumental in winning the nationwide competition, will convene to plan a roadmap of next steps for the city and its initial $257 million in government funding.

With the official announcement in August, Daegu City as Robotics Technopolis serves as a brilliant capstone to Chairman Kim’s ten years of persistent work building out the city’s robotics ecosystem.

With over 100 robotics companies, educational institutions and R&D facilities already resident in Daegu City, and with another 500 projected between 2022 and 2028, the Daegu Robot Technopolis, known officially as the National Robot Innovation Project, might well become the largest robot city in Asia.

The rise of Daegu as Robot Technopolis begins in earnest this November 17 with the annual board meeting and general assembly of the Global Robot Cluster (GRC), founded by Chairman Kim, and built out to now include twenty members from seventeen countries. The GRC and its member delegations, together with Daegu’s Mayor Kwon Young-jin, who was instrumental in winning the nationwide competition, will convene to plan a roadmap of next steps for the city and its initial $257 million in government funding.

The Embedded Vision Summit is coming up May 16–19 in Santa Clara, California. It’s a conference uniquely focused on practical computer vision and visual artificial intelligence (AI), aimed squarely at innovators incorporating vision capabilities in products. One of the great things about being part of the Summit team is seeing trends emerge in the embedded vision space, and the editors at EE Times asked me to share some of the things we’re seeing in 2022.

The first trend that jumped out at me is the tremendous increase in performance and efficiency for embedded vision applications. Interestingly, these increases aren’t just from processors. Certainly, processors are getting faster, often due to a diversity of architectural approaches (a “Cambrian Explosion,” as my colleague Jeff Bier wrote about recently). But algorithms and tools are driving this increase as well. A great example of a practical algorithmic innovation is Edge Impulse’s talk on “Faster Objects, More Objects” (FOMO!), presented by their CTO Jan Jongboom. Similarly, Felix Baum, director of Product Management at Qualcomm, will discuss the company’s latest tools to help developers get the best possible machine learning performance out of their embedded processors.

(By the way, the great thing about this trend is that these performance gains are multiplicative: when you combine efficiency increases in algorithms, tools, and processors — any one of which might be significant on its own — you quickly realize you’re looking at a fantastic year–over–year improvement.)

A second trend is democratization of edge AI by simplifying development. For edge and vision AI to become mainstream, system developers without deep experience must be able to master the technology. This means more use of off–the–shelf models, like the 270+ models available in the OpenVINO Open Model Zoo, featured in the talk by Intel’s Ansley Dunn and Ryan Loney. And it means raising the level of abstraction for developers with low–code/no–code tools, such as those presented by NVIDIA’s Alvin Clark.

A third trend is deployment at scale. How do you get from proof of concept to deployment at scale? Emerging MLops techniques and tools mean that product developers are no longer on their own to figure out thorny problems like version control for training data, as we’ll see in Nicolás Eiris’ talk on AI reproducibility and continuous updates and Rakshit Agrawal’s talk on Kubernetes and containerization for edge vision applications.

The fourth trend concerns reliability and trustworthiness of AI. As AI–enabled systems are deployed more widely, there are more opportunities for mistakes that can have serious consequences. Industry veterans will share their perspectives on how to make AI more trustworthy. Notable examples here are Krishnaram Kenthapadi’s talk on responsible AI and model ops and Robert Laganiere’s talk on sensor fusion. Too, there are important questions to consider about privacy, bias, and ethics in AI. Professor Susan Kennedy from Santa Clara University will present on “Privacy: a Surmountable Challenge for Computer Vision,” followed by an extended audience Q&A session called “Ask the Ethicist: Your Questions about AI Privacy, Bias, and Ethics Answered.”

This is such an exciting time to be involved in edge AI and vision. What trends will you spot at the Summit?

Arm has unveiled an expansion of its ‘Total Solutions for IoT’ roadmap, with two new Corstone subsystems for Cortex-M and Cortex-A processors, and adding more platforms, including Raspberry Pi, to its Arm Virtual Hardware. As part of the new lineup, it has also launched the Arm Cortex-M85 processor, its highest performance Cortex-M to date.

Almost every marketer you talk to in the chip industry talks about massive IoT market opportunities, but the fact remains that integration of processor intellectual property (IP) and building a system-on-chip (SoC) can be challenging. This is even more so where you have increasingly higher performance and security requirements, but don’t necessarily have the development resources to get to market quickly enough.

Hence Arm’s strategy, launched six months ago, is to provide validated and integrated subsystems as part of its total solutions for IoT offer. The first of these was the Corstone-300 solution for keyword recognition launched last year. Now it has added the Corstone-310 solution for voice recognition and Corstone-1000 for cloud native edge devices. Future Corstone products could include vision systems, object recognition, and smart sensor fusion.

Arm’s vice president of IoT and embedded, Mohamed Awad, said that it’s all about simplifying the development. He commented, “Developers are faced with an ever-increasing demand for higher performance, increased security and less complex development flows, all while getting products to market faster than ever. They need more choice, simpler development and more secure processing to continue to scale.”

Hence Arm total solutions for IoT combines hardware IP, platform software, machine learning (ML) models, and tools to simplify development and accelerate product design. The foundation of this is Corstone, a pre-integrated, pre-verified IP subsystem that frees silicon designers to focus their time and efforts on differentiation.

Two new total solutions

The two new subsystems launched as part of its total solutions for IoT portfolio are for:

• Cloud-native edge platform using the new Corstone-1000 for Linux capable design for cloud-native applications.
• Voice recognition using the Corstone-310 based on the new Arm Cortex-M85.

The solution for cloud native edge devices is the first designed for Cortex-A and is based on Corstone-1000. Arm said this total solution makes the power and potential of operating systems like Linux easily available to IoT developers for the first time. It allows application-class workloads to be developed for devices such as smart wearables, gateways, and high-end smart cameras. The Corstone-1000 is Arm SystemReady-IR compliant and features a hardware secure enclave that supports PSA Certified for a higher level of security.

The new total solution for voice recognition is based on the Corstone-310 subsystem. It is pre-integrated with the new Cortex-M85 and the Arm Ethos-U55 to create Arm’s highest ever performance MCU-based design. The solution is targeted at use-cases ranging from smart speakers and thermostats to drones and factory robots. Developers can also take the Corstone-310 and create a whole range of additional high-performance products by combining it with different reference software.

The Arm Cortex-M85 is a natural architectural upgrade path to Armv8-M for applications requiring significantly higher performance. It offers:

• A 30% scalar performance uplift compared to the Cortex-M7.
• Arm Helium technology to support endpoint ML and DSP workloads
• Enhanced security with Arm TrustZone technology. It also includes pointer authentication and branch target identification (PACBTI), a new architectural feature with enhanced software attack threat mitigation to help achieve PSA Certified Level 2, a security baseline for IoT deployments.

Renesas’ EVP & GM for its IoT and infrastructure business unit, Sailesh Chittipeddi, said, “As we expand our high performance and advanced IoT security MCU business, Renesas welcomes the timely introduction of Cortex-M85, providing industry breakthrough performance.”

Also commenting on the new processor, Ricardo De Sa Earp, EVP for the general-purpose microcontroller sub-group at STMicroelectronics, said, “Adding the high-performance and TrustZone capabilities of the Cortex-M85 core to an STM32 MCU is a major opportunity for developers to push the limits for new connected and secure applications and will open up a new range of AI use cases.”

Virtual hardware library now includes NXP, ST, and Raspberry Pi

Arm virtual hardware is designed to enable software development in advance of silicon. It allows the Arm ecosystem to easily adopt cloud-based development and CI/CD, without the need for large custom hardware farms. Arm said hundreds of developers have used Arm virtual hardware to date and based on developer feedback, it is introducing several new virtual devices to broaden the virtual development environment’s appeal. New additions will include Arm virtual hardware for the new Corstone designs as well as seven Cortex-M processors ranging from Cortex-M0 to Cortex-M33. The library is being expanded further with third party hardware from partners including NXP, STMicroelectronics and Raspberry Pi.

A key benefit of extending the virtual hardware to ecosystem devices and a majority of Cortex-M products, it makes it easier for independent software vendors and cloud service providers to take advantage and build upon the large number of Arm-based IoT and embedded devices which are already deployed.

Offering virtual hardware clearly has a lot of support from many in the industry as developers look for faster routes for development and getting a product to market in a timely manner. Qiao Zhao, Head of the PaddlePaddle Product team at Baidu, commented, ““As the IoT continues to evolve, the integration of open-source platforms for deep learning with SoC design platforms will hugely increase the efficiency of smart device development. With the deep integration of the PaddlePaddle (PP) industry-level model library and inference capability with Arm Virtual Hardware, developers can quickly and efficiently deploy the PP-series models, one of the most popular industry-grade models on GitHub, on Cortex-M based hardware, to quickly complete the prototype validation of endpoint AI systems.”

Zach Shelby, co-founder and CEO of Edge Impulse, added, ““Finding the right balance between DSP configuration and model architecture against memory and latency constraints is a big challenge for edge ML developers. By having access to a broader range of models on Arm Virtual Hardware, we can easily estimate performance across a wider spectrum of IoT devices, all in the cloud. This will ultimately provide developers with a faster turnaround and give them the confidence they need to deploy optimized models to edge devices in the field.”

Raspberry Pi CEO, Eben Upton, said, “Offering an easily accessible, virtual version of Raspberry Pi through Arm Virtual Hardware will permit even more developers to test out our technology, solve problems and express themselves through creative projects.”

New open IoT SDK framework

Arm’s Project Centauri was launched a while ago to enable the portability and re-use of software across a range of devices and allow the Cortex-M software ecosystem to coalesce around a consistent set of standards. This includes the Open-CMSIS-Pack, which is already supported by 9,500 microcontrollers and 450 boards, enabling software vendors to easily scale their offerings across all of these devices.

Now, Arm is delivering the first release of the Open IoT SDK Framework as part of Project Centauri. This contains the new Open-CMSIS-CDI software standard, a community driven project hosted in Linaro that defines a Common Device Interface (CDI) for the Cortex-M ecosystem. Eight key industry players are already involved including silicon partners, cloud service providers, ODMs and OEMs.

Arm Cortex-M85, Corstone-310 and Corstone-1000 are available for licensing now and can be accessed in the cloud as part of Arm total solutions for IoT. Arm virtual hardware can be accessed here, and third party hardware is available from partners including NXP (iMX8 Arm Cortex complex), ST Microelectronics (STM32U5 discovery kit) and Raspberry Pi (RPi4).

Is computer vision about to reinvent itself, again?

Ryad Benosman, professor of Ophthalmology at the University of Pittsburgh and an adjunct professor at the CMU Robotics Institute, believes that it is. As one of the founding fathers of event–based vision technologies, Benosman expects neuromorphic vision — computer vision based on event–based cameras — is the next direction computer vision will take.

“Computer vision has been reinvented many, many times,” he said. “I’ve seen it reinvented twice at least, from scratch, from zero.”

Benosman cites a shift in the 1990s from image processing with a bit of photogrammetry to a geometry–based approach, and then today with the rapid change towards machine learning. Despite these changes, modern computer vision technologies are still predominantly based on image sensors — cameras that produce an image similar to what the human eye sees.

According to Benosman, until the image sensing paradigm is no longer useful, it holds back innovation in alternative technologies. The effect has been prolonged by the development

of high–performance processors such as GPUs which delay the need to look for alternative solutions.

“Why are we using images for computer vision? That’s the million–dollar question to start with,” he said. “We have no reasons to use images, it’s just because there’s the momentum from history. Before even having cameras, images had momentum.”

IMAGE CAMERAS

Image cameras have been around since the pinhole camera emerged in the fifth century B.C. By the 1500s, artists built room–sized devices used to trace the image of a person or a landscape outside the room onto canvas. Over the years, the paintings were replaced with film to record the images. Innovations such as digital photography eventually made it easy for image cameras to become the basis for modern computer vision techniques.

Benosman argues, however, .image camera–based techniques for computer vision are hugely inefficient. His analogy is the defense system of a medieval castle: guards positioned around the ramparts look in every direction for approaching enemies. A drummer plays a steady beat, and on each drumbeat, every guard shouts out what they see. Among all the shouting, how easy is it to hear the one guard who spots an enemy at the edge of a distant forest?

The 21^st century hardware equivalent of the drumbeat is the electronic clock signal and the guards are the pixels — a huge batch of data is created and must be examined on every clock cycle, which means there is a lot of redundant information and a lot of unnecessary computation required.

“People are burning so much energy, it’s occupying the entire computation power of the castle to defend itself,” Benosman said. If an interesting event is spotted, represented by the enemy in this analogy, “you’d have to go around and collect useless information, with people screaming all over the place, so the bandwidth is huge… and now imagine you have a complicated castle. All those people have to be heard.”

Enter neuromorphic vision. The basic idea is inspired by the way biological systems work, detecting changes in the scene dynamics rather than analyzing the entire scene continuously. In our castle analogy, this would mean having guards keep quiet until they see something of interest, then shout their location to sound the alarm. In the electronic version, this means having individual pixels decide if they see something relevant.

“Pixels can decide on their own what information they should send, instead of acquiring systematic information they can look for meaningful information — features,” he said. “That’s what makes the difference.”

This event–based approach can save a huge amount of power, and reduce latency, compared to systematic acquisition at a fixed frequency.

“You want something more adaptive, and that’s what that relative change [in event–based vision] gives you, an adaptive acquisition frequency,” he said. “When you look at the amplitude change, if something moves really fast, we get lots of samples. If something doesn’t change, you’ll get almost zero, so you’re adapting your frequency of acquisition based on the dynamics of the scene. That’s what it brings to the table. That’s why it’s a good design.”

Benosman entered the field of neuromorphic vision in 2000, convinced that advanced computer vision could never work because images are not the right way to do it.

“The big shift was to say that we can do vision without grey levels and without images, which was heresy at the end of 2000 — total heresy,” he said.

The techniques Benosman proposed — the basis for today’s event–based sensing — were so different that papers presented to the foremost IEEE computer vision journal at the time were rejected without review. Indeed, it took until the development of the dynamic vision sensor (DVS) in 2008 for the technology to start gaining momentum.

NEUROSCIENCE INSPIRATION

Neuromorphic technologies are those inspired by biological systems, including the ultimate computer, the brain and its compute elements, the neurons. The problem is that no–one fully understands exactly how neurons work. While we know that neurons act on incoming electrical signals called spikes, until relatively recently, researchers characterized neurons as rather sloppy, thinking only the number of spikes mattered. This hypothesis persisted for decades. More recent work has proven that the timing of these spikes is absolutely critical, and that the architecture of the brain is creating delays in these spikes to encode information.

Today’s spiking neural networks, which emulate the spike signals seen in the brain, are simplified versions of the real thing — often binary representations of spikes. “I receive a 1, I wake up, I compute, I sleep,” Benosman explained. The reality is much more complex. When a spike arrives, the neuron starts integrating the value of the spike over time; there is also leakage from the neuron meaning the result is dynamic. There are also around 50 different types of neurons with 50 different integration profiles. Today’s electronic versions are missing the dynamic path of integration, the connectivity between neurons, and the different weights and delays.

“The problem is to make an effective product, you cannot [imitate] all the complexity because we don’t understand it,” he said. “If we had good brain theory, we would solve it — the problem is we just don’t know [enough].”

Today, Bensoman runs a unique laboratory dedicated to understanding the mathematics behind cortical computation, with the aim of creating new mathematical models and replicating them as silicon devices. This includes directly monitoring spikes from pieces of real retina.

For the time being, Benosman is against trying to faithfully copy the biological neuron, describing that approach as old–fashioned.

“The idea of replicating neurons in silicon came about because people looked into the transistor and saw a regime that looked like a real neuron, so there was some thinking behind it at the beginning,” he said. “We don’t have cells; we have silicon. You need to adapt to your computing substrate, not the other way around… if I know what I’m computing and I have silicon, I can optimize that equation and run it at the lowest cost, lowest power, lowest latency.”

PROCESSING POWER

The realization that it’s unnecessary to replicate neurons exactly, combined with the development of the DVS camera, are the drivers behind today’s neuromorphic vision systems. While today’s systems are already on the market, there is still a way to go before we have fully human–like vision available for commercial use.

Initial DVS cameras had “big, chunky pixels,” since components around the photo diode itself reduced the fill factor substantially. While investment in the development of these cameras accelerated the technology, Benosman made it clear that the event cameras of today are simply an improvement of the original research devices developed as far back as 2000. State–of–the–art DVS cameras from Sony, Samsung, and Omnivision have tiny pixels, incorporate advanced technology such as 3D stacking, and reduce noise. Benosman’s worry is whether the types of sensors used today can successfully be scaled up.

“The problem is, once you increase the number of pixels, you get a deluge of data, because you’re still going super fast,” he said. “You can probably still process it in real time, but you’re getting too much relative change from too many pixels. That’s killing everybody right now, because they see the potential, but they don’t have the right processor to put behind it.”

General–purpose neuromorphic processors are lagging behind their DVS camera counterparts. Efforts from some of the industry’s biggest players (IBM Truenorth, Intel Loihi) are still a work in progress. Benosman said that the right processor with the right sensor would be an unbeatable combination.

“[Today’s DVS] sensors are extremely fast, super low bandwidth, and have a high dynamic range so you can see indoors and outdoors,” Benosman said. “It’s the future. Will it take off? Absolutely!”

“Whoever can put the processor out there and offer the full stack will win, because it’ll be unbeatable,” he added.

Despite the long wait and a $2 billion investment before it was ready for prime time, NASA’s newest Mars Rover — the Perseverance — is already making a name for itself.

Perseverance, nicknamed Percy, was first launched into space in July of 2020, but it didn’t land and become active until February of 2021. And since then, Percy has been busy.

With seven payload instruments in tow, two microphones, and 19 cameras, the rover has been exploring the Red Planet looking for evidence of microbial life from the past and signs of habitable environments for the future.

Percy is taking its role seriously, as evidenced by the rover’s most recent, record-breaking spectacle. According to NASA, it has shattered the record for the longest distance driven on Mars by traversing 1,047 feet in one day, largely on its own.

Engineers on earth help guide Percy across rugged terrain but have increasingly allowed the device to rely on its autonomous capabilities to select its route. According to NASA, “Perseverance's auto-navigation system makes 3D maps of the terrain ahead, identifies hazards, and plans a route around any obstacles without additional direction from controllers back on Earth.”

As Gizmodo points out, the biggest development has been how these autonomous tools have been utilized: older versions of rovers, like the Curiosity, had self-driving features, but they had to stop to think. Percy can change course on the fly when it encounters obstacles. It’s this technology, which scientists are calling the “thinking while driving capability” that has enabled the newest rover to cover so much ground.

And there’s more to come. Percy’s destination is a river delta at the end of a three-mile journey across the Jezero Crater. NASA hopes Percy can collect rocks and soil samples that might provide signs of ancient life.

In March, Chipotle introduced Chippy, an AI-powered robotic arm that makes intentionally imperfect tortilla chips; some with slightly more salt, others with a more distinct tang of lime. And Chippy isn’t the only robot being put to work; Cecilia.ai, a mechanical mixologist, is being implemented in bars around the world to serve up the perfect margarita while chatting with customers using conversational AI. 

Since the mid-2010s, the world has been advancing Industry 4.0, which is a combination of artificial intelligence (AI), additive manufacturing, and the Internet of Things (IoT). Experts argue that the COVID-19 pandemic accelerated the shift to Industry 5.0 and that soon AI-powered platforms and robots will largely take on monotonous tasks that no longer require human labor. 

So how do robots learn to fulfill these tasks? And can they expand their knowledge on their own?

What Is Artificial Intelligence (AI)?

As far back as the 1950s, Alan Turing, often considered the father of computer science, asked, “Can machines think?” Since then, AI has been defined as the use of machines and computers to mimic human behavior like the problem-solving capabilities required to fulfill tasks that once required human intelligence. 

Combining computer science and datasets, AI algorithms use data to make predictions and fulfill tasks. Deep learning and machine learning are both subsets of AI, although often used interchangeably, and have been used for speech recognition, customer service, stock trading, and more.

Some of the more commonly-known uses of AI include Siri and Alexa, self-driving cars, email spam filters, and even Netflix recommendations.  

How Do Robots Learn?

In the long term, scientists and engineers want to create AI that can take on tasks ranging from driving people to and from locations in a taxi to stocking grocery shelves. 

Josh Tenenbaum, a psychologist at MIT in Cambridge, says AI-powered robots should be able to “interact in the full human world like C-3PO.” But to accomplish this, advanced machine learning is required. 

Training is often the hardest part of developing an AI-powered system, as it calls for time, a plethora of resources, and suitable data. The CTO of FruitCast, an agricultural AI startup, said it takes real-world training and examples to properly train AI because robots aren’t very smart — until you make them smart. 

To develop an AI system, the computer or robot first gathers data about a specific task or situation through human input and sensors. Using other stored data, the system then decides which action, based on the scenario, will be most successful. 

Because the system can only use the data it has available at any given time, if it is asked to do something it isn’t equipped to do it may fail. Think of these programmed AIs like an automated message when you call the doctor's office to make an appointment: If you say something the system is unfamiliar with, it usually asks you to repeat yourself or suggests an alternative. 

It’s true that these bots can definitely be trained to work on an assembly line, but this doesn’t mean that robots aren’t simply programmed to fulfill tasks. 

Do Robots Learn or Are They Merely Programmed?  

Robots can learn… to an extent. For example, robotic vacuums often have the capability to learn the layout of a room. There are also social robots currently employed as aids for the elderly that are coded to help them clean, get in and out of bed, and retrieve meals. However, these bots are only programmed to do specific tasks. 

Machine learning often uses a “neural network,” which is a set of data used for training. This method is inspired by the workings of the human brain and functions by giving the system a dataset and the solution, then allowing it to study the data. Then, once it is “trained,” it is tested without being given the solution until it correctly identifies the answer nearly 100% of the time. 

Like the human brain, natural intelligence is complex and ultimately required to push AI to its full potential. “We do know that the brain contains billions and billions of neurons, and that we think and learn by establishing electrical connections between different neurons,” writes HowStuffWorks. “But we don't know exactly how all of these connections add up to higher reasoning or even low-level operations.”

This is why many scientists are focused on humanoid robots, as fully operational AIs need both innate and programmed abilities that come with more human-like intelligence. It must be trainable and learn from its own experiences. 

Humanoid robots often have actuators, which allow them to sense their environment. But these bots are still coded to fulfill certain tasks, and while the capacity to interact with humans is still widely limited, AI allows humanoids to understand commands, answer questions, and even respond sarcastically or use slang. 

So, depending on your definition of learning, robots can “learn” in some capacity.