Differences

This shows you the differences between two versions of the page.

--- en:iot-reloaded:iot_data_analysis [2024/11/29 10:23] – pczekalski
+++ en:iot-reloaded:iot_data_analysis [2025/05/17 08:56] (current) – agrisnik
@@ Line 1: / Line 1: @@
 ====== IoT Data Analysis ======
-IoT systems, in their essence, are built to act as a tool for getting better insights into different processes and systems in order to make better decisions. The insights are provided by measuring the statuses of the systems or process elements represented by data. Unfortunately, without properly interpreting the data content, they turn into useless bits and bytes. Therefore, providing a means for understanding data is an essential property of a modern IoT system.
+IoT systems are built to provide better insights into different processes and systems to make better decisions. The insights are provided by measuring the statuses of the systems or process elements represented by data. Unfortunately, the bits and bytes become useless without adequately interpreting the data content. Therefore, providing a means for understanding data is an essential property of a modern IoT system.
 Today, IoT systems produce a vast amount of data, which is very hard to use manually. Thanks to modern hardware and software developments, it is possible to develop fully or semi-automated systems for data analysis and interpretation, which may go further into decision-making and acting according to the decisions.
-As various resources have stated, IoT in most cases, complies with the so-called big 5Vs of Big Data, where just one correspondence is needed to solve a Big Data problem. As has been explained by Jain et al. ((Jain, A., Mittal, S., Bhagat, A., Sharma, D.K. (2023). Big Data Analytics and Security Over the Cloud: Characteristics, Analytics, Integration and Security. In: Srivastava, G., Ghosh, U., Lin, J.CW. (eds) Security and Risk Analysis for Intelligent Edge Computing. Advances in Information Security, vol 103. Springer, Cham. https://doi.org/10.1007/978-3-031-28150-1_2)) Big Data might be of different forms, volumes and structures, and in general, the 5Vs i.e. Volume, Variety, Veracity, Velocity and Value might be interpreted as follows:
+As various resources have stated, IoT, in most cases, complies with the so-called big 5Vs of Big Data, where just one correspondence is needed to solve a Big Data problem. As has been explained by Jain et al. ((Jain, A., Mittal, S., Bhagat, A., Sharma, D.K. (2023). Big Data Analytics and Security Over the Cloud: Characteristics, Analytics, Integration and Security. In: Srivastava, G., Ghosh, U., Lin, J.CW. (eds) Security and Risk Analysis for Intelligent Edge Computing. Advances in Information Security, vol 103. Springer, Cham. https://doi.org/10.1007/978-3-031-28150-1_2)) Big Data might be of different forms, volumes and structures, and in general, the 5Vs, e.g. Volume, Variety, Veracity, Velocity and Value might be interpreted as follows:
 === Volume ===
@@ Line 11: / Line 11: @@
 === Variety ===
-As Jain explained, big data is highly heterogeneous in terms of source, kind, and nature. Having different systems, processes, sensors, and other data sources, variety is usually a distinctive feature of practical IoT systems. For instance, a system of intelligent office buildings would need data from a building management system, appliances and independent sensors, and external sources like weather stations or forecasts from appropriate external weather forecast APIs (Application programming interfaces). Additionally, the given system might require historical data from other sources, like XML documents, CSV files or other sources, diversifying the sources even more.
+Jain explained that Big Data is highly heterogeneous regarding source, kind, and nature. Having different systems, processes, sensors, and other data sources, variety is usually a distinctive feature of practical IoT systems. For instance, a system of intelligent office buildings would need data from a building management system, appliances and independent sensors, and external sources like weather stations or forecasts from appropriate external weather forecast APIs (Application programming interfaces). Additionally, the given system might require historical data from other sources, like XML documents, CSV files or other sources, diversifying the sources even more.
 === Veracity ===
@@ Line 19: / Line 19: @@
 === Velocity ===
-Data velocity characterises the data bound to the time and its importance during a specific period or at a particular time instant. A good example might be any real-time system like an industrial process control system, where reactions or decisions must be made during a fixed period of time, requiring data at particular time instants. In this case, data has a flow nature of a particular density.
+Data velocity characterises the data bound to the time and its importance during a specific period or at a particular time instant. A good example might be any real-time system like an industrial process control system, where reactions or decisions must be made during a fixed period, requiring data at particular time instants. In this case, data has a flow nature of a specific density.
 === Value ===
-Since the IoT systems and their data analysis subsystems are built to add value to their owner, the costs of the development and ownership should exceed the returned value. The system is of low or no value if it does not apply.
+Since IoT systems and their data analysis subsystems are built to add value to their owners, the development and ownership costs should not exceed the returned value. A system is of low or no value if it does not apply.
 ====== ======
-Dealing with big data requires specific hardware and software infrastructure. While there is a certain number of typical solutions and a lot more customise, some of the most popular are explained here:
+Dealing with Big Data requires specific hardware and software infrastructure. While there is a certain number of typical solutions and a lot more customised, some of the most popular are explained here:
 === Relational DB-based systems ===
 Those systems are based on well-known relational data models and appropriate database management systems like MS SQL Server, Oracle Server, MySQL, etc. There are some advantageous features of those systems, for instance:
-  * Advantages of SQL (Structured Querying Language): enabling easy manipulation of the data while maintaining a relatively good expressiveness of the data model;
+  * Advantages of SQL (Structured Querying Language): enabling easy data manipulation while maintaining a relatively good expressiveness of the data model.
-  * A well-designed set of software tools and interfaces enabling integration with a large number of different systems;
+  * A well-designed set of software tools and interfaces enabling integration with many different systems.
   * A lot of built-in data processing routines (stored procedures) provide higher development productivity.
   * Enables asynchronous reactions to events by triggering internal events.
   * Data reading might be scaled out using multiple entities, while writing might be scaled up using more productive servers.
-Unfortunately, scaling out data writing is not always possible and is usually supported at a high cost for software products.
+Unfortunately, scaling out data writing is not always possible and is usually supported at a high cost for software products (figure 1).
 <figure RelationalDBMS>
-{{ :en:iot-reloaded:dbs2.png?400 | Relational DBMS scaling options}}
+{{ :en:iot-reloaded:dbs2.png?400 | Relational DBMS Scaling Options}}
-<caption>Relational DBMS scaling options</caption>
+<caption>Relational DBMS Scaling Options</caption>
 </figure>
 === Complex Event Processing (CEP) systems ===
-CEP systems are very application-tailored, enabling significant productivity at a reasonable cost. High productivity is usually needed for processing data streams, such as voice or video. Maintaining a limited time window for data processing is possible, which is relevant for systems that are close to real-time.
+CEP systems are very application-tailored, enabling significant productivity at a reasonable cost. High productivity is usually needed for processing data streams, such as voice or video. Maintaining a limited time window for data processing is possible, which is relevant for systems close to real-time (figure {{ref>CEP_systems}}).
 Some of the most common drawbacks to be considered are:
   * It might be scaled up only by introducing higher productivity hardware, which is limited by the application-specific design. To some extent, the design might be more flexible if microservices and containerisation are applied.
-  * Due to the factors mentioned above and the complexity, the maintenance costs are usually higher than a universal design.
+  * Due to the factors mentioned above and the complexity, the maintenance costs are usually higher than a universal design (figure 2).
-<figure CEP systems>
+<figure CEP_systems>
-{{ :en:iot-reloaded:ceps.png?400 | CEP systems}}
+{{ :en:iot-reloaded:ceps.png?400 | CEP Systems}}
-<caption>CEP systems</caption>
+<caption>CEP Systems</caption>
 </figure>
 === NoSQL systems ===
-As the name suggests, the main characteristic is higher flexibility in data models, which overcomes the limitations of highly structured relational data models. NoSQL systems are usually distributed, where the distribution is the primary tool to enable supreme flexibility. In IoT systems, software typically gets older faster than hardware, which requires the maintenance of many versions of communication protocols and data formats to ensure back compatibility. Another reason is the variety of hardware suppliers, where some protocols or data formats are specific to the given vendor.
+As the name suggests, the main characteristic is higher flexibility in data models, which overcomes the limitations of highly structured relational data models (figure {{ref>NoSQL_systems}}). NoSQL systems are usually distributed, where the distribution is the primary tool to enable supreme flexibility. In IoT systems, software typically gets older faster than hardware, which requires the maintenance of many versions of communication protocols and data formats to ensure back compatibility. Another reason is the variety of hardware suppliers, where some protocols or data formats are specific to the given vendor.
-It also provides a means for scalability out and up, enabling high future tolerance and resilience. A typical approach is to use a key-value or key-document approach, where a unique key indexes incoming data blocks or documents (JSON, for instance).
+It also provides a means for scalability out and up, enabling high future tolerance and resilience. A typical approach uses a key-value or key-document approach, where a unique key indexes incoming data blocks or documents (JSON, for instance).
-Some other designs might extend the SQL data models by others – object models, graph models, or the mentioned key-value models, providing highly purpose-driven and, therefore, productive designs. However, the complexity of the design raises problems of data integrity as well as the complexity of maintenance.
+Some other designs might extend the SQL data models by others – object models, graph models, or the mentioned key-value models, providing highly purpose-driven and, therefore, productive designs. However, the complexity of the design raises problems of data integrity as well as the complexity of maintenance (figure 3).
-<figure NoSQL systems>
+<figure NoSQL_systems>
-{{ :en:iot-reloaded:keyvalue.png?400 | NoSQL systems}}
+{{ :en:iot-reloaded:keyvalue.png?400 | NoSQL Systems}}
-<caption>NoSQL systems</caption>
+<caption>NoSQL Systems</caption>
 </figure>
@@ Line 69: / Line 69: @@
 This is probably the most productive type of system, providing high flexibility, productivity and scalability. Because these systems are designed to operate in servers RAM, the in-memory data grids are the best choice for data preprocessing in IoT systems due to their high productivity and ability to scale dynamically depending on actual workloads. They provide all the benefits of the CEP and Relational systems, adding a scale-out functionality for data writing. There are only two major drawbacks – limited RAM and high development costs. Some examples of available solutions:
-  * Hazelcast    http://hazelcast.org (Uses in-memory NoSQL DB)
+  * Hazelcast ((http://hazelcast.org)) (Uses in-memory NoSQL DB)
-  * JBOSS Inﬁnispan https://infinispan.org/ (Appache based key-value store)
+  * JBOSS Inﬁnispan ((https://infinispan.org/)) (Apache-based key-value store)
-  * IBM eXtreme Scale https://www.ibm.com/products/ibm-websphere-extreme-scale
+  * IBM eXtreme Scale ((https://www.ibm.com/products/ibm-websphere-extreme-scale))
-  * Gigaspace XAP Elastic caching edition www.gigaspaces.com/xap-in-memory-caching-scaling/datagrid (Transactions-based microservices)
+  * Gigaspace XAP Elastic caching edition ((www.gigaspaces.com/xap-in-memory-caching-scaling/datagrid)) (Transactions-based microservices)
-  * Oracle Coherence  www.oracle.com/technetwork/middleware/coherence
+  * Oracle Coherence  ((www.oracle.com/technetwork/middleware/coherence))
-  * Terracotta enterprise suite www.terracotta.org/products/enterprise-suite
+  * Terracotta enterprise suite ((www.terracotta.org/products/enterprise-suite))
-  * Pivotal Gemﬁre    pivotal.io/big-data/pivotal-gemﬁre
+  * Pivotal Gemﬁre    ((https://pivotal.io/big-data/pivotal-gemﬁre))
-======  ======
+<WRAP excludefrompdf>
 This chapter is devoted to the main groups of algorithms for numerical data analysis and interpretation, covering both mathematical foundations and application specifics in the context of IoT. The chapter is split into the following subchapters:
@@ Line 88: / Line 88: @@
   * [[en:iot-reloaded:Introduction_to_time_series_analysis]],
   * [[en:iot-reloaded:Hints_for_further_readings_on_AI]].
+</WRAP>

en/iot-reloaded/iot_data_analysis.1732875812.txt.gz · Last modified: 2024/11/29 10:23 by pczekalski