Loading Data into PathWave Manufacturing Analytics
Application Notes
With the emergence of Industry 4.0, the deluge of devices, test software, and test equipment rushing to support the digital transformation of the factory has increased rapidly over the years. Often, each of these has their own formats, or a wrinkle of an existing industry standard, and yet, data out of these varied sources and systems was expected to flow into Enterprise Resource Planning (ERP) and analytic systems in copious amounts. Hence, the ability to get the right data anytime or at the right time becomes even more crucial. One can find that there are many articles discussing data storage systems (Data Lakes, HDFS, HBase) and data retrieval and query solutions (SQL, Impala, Hive). However, it’s not easy to find topics related to device and test equipment data collection mechanism. There are 2 main challenges that sorely needs solutions to, namely data parsing and data pipeline.