repository open issue suggest edit

Contents

Datasets

API

Dataset

class heedy.datasets.Dataset(h, x=None, **kwargs)

Bases: object

Heedy is capable of taking several separate unrelated timeseries, and based upon the chosen interpolation method, putting them all together to generate tabular data centered about either another timeseries’ datapoints, or based upon time intervals. The underlying issue that Datasets solve is that in Heedy, timeseries are inherently unrelated. In most data stores, such as standard relational (SQL) databases, and even excel spreadsheets, data is in tabular form. That is, if we have measurements of temperature in our house and our mood, we have a table:

Mood Rating	Room Temperature (F)
7	73
3	84
5	79

The benefit of having such a table is that it is easy to perform data analysis. You know which temperature value corresponds to which mood rating. The downside of having such tables is that Mood Rating and Room Temperature must be directly related - a temperature measurement must be made each time a mood rating is given. Heedy has no such restrictions. Mood Rating and Room Temperature can be entirely separate sensors, which update data at their own rate. In Heedy, each timeseries can be inserted with any timestamp, and without regard for any other data. This separation of Timeseries makes data require some preprocessing and interpolation before it can be used for analysis. This is the purpose of the Dataset query. Heedy can put several streams together based upon chosen transforms and interpolators, returning a tabular structure which can readily be used for ML and statistical applications. There are two types of dataset queries :T-Dataset:

T-Dataset: A dataset query which is generated based upon a time range. That is, you choose a time range and a time difference between elements of the dataset, and that is used to generate your dataset.

Timestamp	Room Temperature (F)
1pm	73
4pm	84
8pm	79

If I were to generate a T-dataset from 12pm to 8pm with dt=2 hours, using the interpolator “closest”, I would get the following result:

Timestamp	Room Temperature (F)
12pm	73
2pm	73
4pm	84
6pm	84
8pm	79

The “closest” interpolator happens to return the datapoint closest to the given timestamp. There are many interpolators to choose from (described later). Hint: T-Datasets can be useful for plotting data (such as daily or weekly averages).

X-Dataset

X-datasets allow to generate datasets based not on evenly spaced timestamps, but based upon values of a timeseries. Suppose you have the following data:

Timestamp	Mood Rating		Timestamp	Room Temperature (F)
1pm	7		2pm	73
4pm	3		5pm	84
11pm	5		8pm	81
			11pm	79

An X-dataset with X=Mood Rating, and the interpolator “closest” on Room Temperature would generate:

Mood Rating	Room Temperature (F)
7	73
3	84
5	79

Interpolators

Interpolators are special functions which specify how exactly the data is supposed to be combined into a dataset. Any PipeScript script can be used as an interpolator, including “sum”, “count” and other transforms. By default, the “closest” interpolator is used, which simply returns the datapoint closest to the reference timestamp.

add(key, ts, **kwargs)

Adds the given timeseries to the query construction. Unless an interpolator is specified, “closest” will be used. You can insert a merge query instead of a timeseries:

d = Dataset(h, t1="now-1h",t2="now",dt=10)
d.add("temperature","timeseriesid")
d.add("steps",h.objects["timeseriesid2"])
m = Merge(h)
m.add("id1")
m.add("id2")
d.add("altitude",m)
result = d.run()

run()

Runs the dataset query, and returns the result

Merge

class heedy.datasets.Merge(h)

Bases: object

Merge represents a query which allows to merge multiple timeseries into one when reading, with all the data merged together by increasing timestamp. The merge query is used as a constructor-type object:

m = Merge(h)
m.add("objectid",t1="now-10h")
m.add(h.objects["objectid2"],t1="now-10h")
result = m.run()

previous

Users

next

Notifications