Datasets
Contents
Datasets¶
API¶
Dataset¶
- class heedy.datasets.Dataset(h, x=None, **kwargs)[source]¶
Bases:
object
Heedy is capable of taking several separate unrelated timeseries, and based upon the chosen interpolation method, putting them all together to generate tabular data centered about either another timeseries’ datapoints, or based upon time intervals. The underlying issue that Datasets solve is that in Heedy, timeseries are inherently unrelated. In most data stores, such as standard relational (SQL) databases, and even excel spreadsheets, data is in tabular form. That is, if we have measurements of temperature in our house and our mood, we have a table:
Mood Rating
Room Temperature (F)
7
73
3
84
5
79
The benefit of having such a table is that it is easy to perform data analysis. You know which temperature value corresponds to which mood rating. The downside of having such tables is that Mood Rating and Room Temperature must be directly related - a temperature measurement must be made each time a mood rating is given. Heedy has no such restrictions. Mood Rating and Room Temperature can be entirely separate sensors, which update data at their own rate. In Heedy, each timeseries can be inserted with any timestamp, and without regard for any other data. This separation of Timeseries makes data require some preprocessing and interpolation before it can be used for analysis. This is the purpose of the Dataset query. Heedy can put several streams together based upon chosen transforms and interpolators, returning a tabular structure which can readily be used for ML and statistical applications. There are two types of dataset queries :T-Dataset:
T-Dataset: A dataset query which is generated based upon a time range. That is, you choose a time range and a time difference between elements of the dataset, and that is used to generate your dataset.
Timestamp
Room Temperature (F)
1pm
73
4pm
84
8pm
79
If I were to generate a T-dataset from 12pm to 8pm with dt=2 hours, using the interpolator “closest”, I would get the following result:
Timestamp
Room Temperature (F)
12pm
73
2pm
73
4pm
84
6pm
84
8pm
79
The “closest” interpolator happens to return the datapoint closest to the given timestamp. There are many interpolators to choose from (described later). Hint: T-Datasets can be useful for plotting data (such as daily or weekly averages).
- X-Dataset
X-datasets allow to generate datasets based not on evenly spaced timestamps, but based upon values of a timeseries. Suppose you have the following data:
Timestamp
Mood Rating
Timestamp
Room Temperature (F)
1pm
7
2pm
73
4pm
3
5pm
84
11pm
5
8pm
81
11pm
79
- An X-dataset with X=Mood Rating, and the interpolator “closest” on Room Temperature would generate:
Mood Rating
Room Temperature (F)
7
73
3
84
5
79
- Interpolators
Interpolators are special functions which specify how exactly the data is supposed to be combined into a dataset. Any PipeScript script can be used as an interpolator, including “sum”, “count” and other transforms. By default, the “closest” interpolator is used, which simply returns the datapoint closest to the reference timestamp.
- add(key, ts, **kwargs)[source]¶
Adds the given timeseries to the query construction. Unless an interpolator is specified, “closest” will be used. You can insert a merge query instead of a timeseries:
d = Dataset(h, t1="now-1h",t2="now",dt=10) d.add("temperature","timeseriesid") d.add("steps",h.objects["timeseriesid2"]) m = Merge(h) m.add("id1") m.add("id2") d.add("altitude",m) result = d.run()
Merge¶
- class heedy.datasets.Merge(h)[source]¶
Bases:
object
Merge represents a query which allows to merge multiple timeseries into one when reading, with all the data merged together by increasing timestamp. The merge query is used as a constructor-type object:
m = Merge(h) m.add("objectid",t1="now-10h") m.add(h.objects["objectid2"],t1="now-10h") result = m.run()