Intraday performance report when using 1 minute data?

[kvak]

Q0017.txt
Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: (Windows Version or $ uname --all)
  4.10.0-40-generic #44~16.04.1-Ubuntu SMP Thu Nov 9 15:37:44 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux * Python Version: $ python --version
Python 3.5.2
• Python Bitness: $ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))'
64
• How did you install Zipline: (pip, conda, or other (please explain))
  pip
• Python packages: $ pip freeze or $ conda list

alembic==0.9.6
bcolz==0.12.1
Bottleneck==1.2.1
certifi==2017.11.5
chardet==3.0.4
click==6.7
contextlib2==0.5.5
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.27.3
decorator==4.1.2
empyrical==0.3.3
idna==2.6
intervaltree==2.1.0
Logbook==1.1.0
lru-dict==1.1.6
Mako==1.0.7
MarkupSafe==1.0
matplotlib==2.1.0
multipledispatch==0.4.9
networkx==2.0
numexpr==2.6.4
numpy==1.13.3
pandas==0.18.1
pandas-datareader==0.5.0
patsy==0.4.1
pyparsing==2.2.0
python-dateutil==2.6.1
python-editor==1.0.3
pytz==2017.3
requests==2.18.4
requests-file==1.4.2
requests-ftp==0.3.1
scipy==1.0.0
six==1.11.0
sortedcontainers==1.5.7
SQLAlchemy==1.1.15
statsmodels==0.8.0
tables==3.4.2
toolz==0.8.2
urllib3==1.22
zipline==1.1.1

Now that you know a little about me, let me tell you about the issue I am
having:

Description of Issue

• What did you expect to happen?
  Intraday performance report
• What happened instead?
  Daily performance report

The ingestion step works fine:

zipline ingest -b ingester
entering machina.  tuSymbols= ('Q0017',)
about to return ingest function
entering ingest and creating blank dfMetadata
dfMetadata <class 'pandas.core.frame.DataFrame'>
<bound method NDFrame.describe of   start_date   end_date auto_close_date symbol
0 1970-01-01 1970-01-01      1970-01-01   None>
S= Q0017 IFIL=/merged_data/Q0017.csv
read_csv dfData <class 'pandas.core.frame.DataFrame'> length 7717 2017-06-15 22:00:00
start_date <class 'pandas.tslib.Timestamp'> 2017-06-15 22:00:00 None
end_date <class 'pandas.tslib.Timestamp'> 2017-06-23 20:00:00 None
ac_date <class 'pandas.tslib.Timestamp'> 2017-06-24 20:00:00 None
liData <class 'list'> length 1
Now calling minute_bar_writer
returned from minute_bar_writer
calling asset_db_writer
dfMetadata <class 'pandas.core.frame.DataFrame'>
           start_date            end_date     auto_close_date symbol exchange
0 2017-06-15 22:00:00 2017-06-23 20:00:00 2017-06-24 20:00:00  Q0017      ICE
symbol_map <class 'pandas.core.series.Series'>
returned from asset_db_writer
calling adjustment_writer
returned from adjustment_writer
now leaving ingest function

So I try to run this toy example:

from zipline.api import symbol, get_datetime, record, order_target, history
from pytz import timezone



def initialize(context):
    context.contract  = symbol("Q0017")
    context.i         = 0
    
    
def handle_data(context, data):
    context.i += 1
    if context.i < 30:
        return

    # Compute averages
    # history() has to be called with the same params
    # from above and returns a pandas dataframe.
    short_mavg = data.history(context.contract, 'close', 10, '1m').mean()
    long_mavg  = data.history(context.contract, 'close', 30, '1m').mean()

    # Trading logic
    if short_mavg > long_mavg:
        # order_target orders as many shares as needed to
        # achieve the desired number of shares.
        order_target(context.contract, 100)
    elif short_mavg < long_mavg:
        order_target(context.contract, 0)

    # Save values for later inspection
    record(Q0017      = data.current(context.contract, "close"),
           short_mavg = short_mavg,
           long_mavg  = long_mavg)




# Note: this function can be removed if running
# this algorithm on quantopian.com
def analyze(context=None, results=None):
    import matplotlib.pyplot as plt
    import logbook
    logbook.StderrHandler().push_application()
    log = logbook.Logger('Algorithm')

    fig = plt.figure()
    ax1 = fig.add_subplot(211)
    results.portfolio_value.plot(ax=ax1)
    ax1.set_ylabel('Portfolio value (USD)')

    ax2 = fig.add_subplot(212)
    ax2.set_ylabel('Price (USD)')

    # If data has been record()ed, then plot it.
    # Otherwise, log the fact that no data has been recorded.
    if ('Q0017' in results and 'short_mavg' in results and
            'long_mavg' in results):
        results['Q0017'].plot(ax=ax2)
        results[['short_mavg', 'long_mavg']].plot(ax=ax2)

        trans = results.ix[[t != [] for t in results.transactions]]
        buys = trans.ix[[t[0]['amount'] > 0 for t in
                         trans.transactions]]
        sells = trans.ix[
            [t[0]['amount'] < 0 for t in trans.transactions]]
        ax2.plot(buys.index, results.short_mavg.ix[buys.index],
                 '^', markersize=10, color='m')
        ax2.plot(sells.index, results.short_mavg.ix[sells.index],
                 'v', markersize=10, color='k')
        plt.legend(loc=0)
    else:
        msg = 'Q0017, short_mavg & long_mavg data not captured using record().'
        ax2.annotate(msg, xy=(0.1, 0.5))
        log.info(msg)

    plt.show()

And it runs. but the performance report:

zipline run -f my_first_backtest.py --bundle ingester --data-frequency minute -s 2017-06-15 -e 2017-06-23
entering machina.  tuSymbols= ('Q0017',)
about to return ingest function
[2017-12-04 23:31:30.040649] WARNING: Loader: Refusing to download new benchmark data because a download succeeded at 2017-12-04 23:19:05.652054+00:00.
[2017-12-04 23:31:33.389892] INFO: Performance: Simulated 7 trading days out of 7.
[2017-12-04 23:31:33.390008] INFO: Performance: first open: 2017-06-15 13:31:00+00:00
[2017-12-04 23:31:33.390093] INFO: Performance: last close: 2017-06-23 20:00:00+00:00
                           Q0017  algo_volatility  algorithm_period_return  \
2017-06-15 20:00:00+00:00    NaN              NaN                 0.000000   
2017-06-16 20:00:00+00:00  47.29         0.000104                -0.000009   
2017-06-19 20:00:00+00:00  46.81         0.002406                -0.000276   
2017-06-20 20:00:00+00:00  45.84         0.006135                -0.001100   
2017-06-21 20:00:00+00:00  44.78         0.012046                -0.002896   
2017-06-22 20:00:00+00:00  45.29         0.013814                -0.002144   
2017-06-23 20:00:00+00:00  45.61         0.012915                -0.002037

has only 7 rows (one per day). Given 7717 intraday bars, I would expect
to see one row per bar (7717 of them).

[imkoukou]

The problem is in algorithm.py, you can find the following main loop for all bars:
for perf in self.get_generator():
perfs.append(perf)

        # convert perf dict to pandas dataframe
        daily_stats = self._create_daily_stats(perfs)
        self.analyze(daily_stats)
        
        return daily_stats

the daily_stats keeps all information for analyze() results parameter. Inside the self._create_daily_stats you can find results data is only recorded by every day.
[ My solution ]
Add a _create_minute_stats() fuction to replace the _create_daily_stats() function.
You can just copy the _create_daily_stats function definition, and replace all "daily" string to "minute". I tested, that works.
Then in the loop "for perf in self.get_generator():" use the minute stats like:

        minute_stats = self._create_minute_stats(perfs)
        self.analyze(minute_stats) 
        
       return minute_stats

[apoorvkalra]

@imkoukou could you place an edited algorithm.py file here?
i tried replacing all the daily string to minute string and end up getting no performance data at all..

[imkoukou]

@apoorvkalra Hi. I updated the edited algorithm.py I am using and the original one. You can compare them to see what I changed.

The displayed report depends on what is returned by def run() function in algorithm.py.
I would like to return the daily report, cause volume of minute data report is too annoying.
It would be displayed like this....

algorithm.zip

[studyquant]

it doesn't work in my zipline 1.2.0, python 3.5 . since the perf variable has no
'minute_perf' key.

`
try:
perfs = []
self.get_generator()
for perf in self.get_generator():
perfs.append(perf)

        # convert perf dict to pandas dataframe


        daily_stats = self._create_daily_stats(perfs)
        if self.sim_params.data_frequency == 'daily':
            self.analyze(daily_stats)

        minute_stats = self._create_minute_stats(perfs)
        # Revised by studyquant
        if self.sim_params.data_frequency == 'minute':
            minute_stats = self._create_minute_stats(perfs)
            self.analyze(minute_stats)
    finally:
        self.data_portal = None

`

`
for perf in perfs:
if 'minute_perf' in perf:
# print("perf is:\n",perf)

            perf['minute_perf'].update(
                perf['minute_perf'].pop('recorded_vars')
            )
            perf['minute_perf'].update(perf['cumulative_risk_metrics'])

            daytime = perf['period_start'].strftime("%a")
            # Only analyze day is in the CustomBusinessDay of our calenday.
            if (daytime in workDayStrList):
                minute_perfs.append(perf['minute_perf'])
        else:
            self.risk_report = perf

    minute_dts = pd.DatetimeIndex(
        [p['period_close'] for p in minute_perfs], tz='UTC'
    )

`

[imkoukou]

@studyquant

1. Have you done the correct setting for minute csv data, including calendar setting? In the last version of Zipline, the minute data process have been included, which is much easier to configure for minute data simulation.
2. Did you run the script with minute data_frequency, and the specified start time and end time is within the csv data range

[studyquant]

Dear imkoukou:
thank you for your reply. My data setting and calendar setting is current. And now I have updated the zipline into 1.3.0 version.

The panel like this:

date                                                                      
2018-02-05 17:48:00  7170.000000  7171.000000  7170.000000  7170.990234   
2018-02-05 17:49:00  7131.990234  7171.000000  7170.990234  7131.990234   
2018-02-05 17:50:00  7120.000000  7137.359863  7132.000000  7120.020020   
2018-02-05 17:51:00  7113.000000  7121.000000  7120.040039  7113.000000   
2018-02-05 17:52:00  7113.000000  7122.000000  7113.000000  7121.990234   

volume  
date                            
2018-02-05 17:48:00   3.425961  
2018-02-05 17:49:00   5.209975  
2018-02-05 17:50:00  14.767619  
2018-02-05 17:51:00  18.237879  
2018-02-05 17:52:00  22.768671  
<class 'pandas.core.panel.Panel'>
Dimensions: 1 (items) x 72277 (major_axis) x 5 (minor_axis)
Items axis: BTC to BTC
Major_axis axis: 2018-02-05 17:48:00+00:00 to 2018-03-27 22:24:00+00:00
Minor_axis axis: low to volume

in the zipline/ algorithm.py file, I have add a function in class

        # create minute and cumulative stats dataframe
        minute_perfs = []
        workDayStrList = self.trading_calendar.day.weekmask.split(" ")
        # TODO: the loop here could overwrite expected properties
        # of minute_perf. Could potentially raise or log a
        # warning.
        # perfDF = pd.DataFrame(perfs)
        # print("daily stats perfs are:\n",perfDF.head(),"\n...\n",perfDF.tail())

        for perf in perfs:
            if 'minute_perf' in perf:
                # print("perf is:\n",perf)
    
                perf['minute_perf'].update(
                    perf['minute_perf'].pop('recorded_vars')
                )
                perf['minute_perf'].update(perf['cumulative_risk_metrics'])

                daytime = perf['period_start'].strftime("%a")
                # Only analyze day is in the CustomBusinessDay of our calenday.
                if (daytime in workDayStrList):
                    minute_perfs.append(perf['minute_perf'])
            else:
                self.risk_report = perf

        minute_dts = pd.DatetimeIndex(
            [p['period_close'] for p in minute_perfs], tz='UTC'
        )
       
        minute_stats = pd.DataFrame(minute_perfs, index=minute_dts)

        return minute_stats

and revisde

`        try:
            perfs = []
            for perf in self.get_generator():
                perfs.append(perf)

            # convert perf dict to pandas dataframe
            daily_stats = self._create_daily_stats(perfs)
            if self.sim_params.data_frequency == 'daily':
                self.analyze(daily_stats)

            ### Revised by me ###
            # user srcript analyze is executed here:
            # ### user file : analyze(context=None, results=None)
            if self.sim_params.data_frequency == 'minute':
                minute_stats = self._create_minute_stats(perfs)
                self.analyze(minute_stats)
        finally:
            self.data_portal = None

        # return None
        # return daily_stats shall display the daily report after simulation even if in minute frequency. 
        # return daily_stats
        # return minute_stats shall display the minute report after simulation, NA for daily frequency.
        return minute_stats
        `

it does not display introday report.
since in _create_minute_stats(self, perfs) function:

`        for perf in perfs:
            if 'minute_perf' in perf:
                # print("perf is:\n",perf)
    
                perf['minute_perf'].update(
                    perf['minute_perf'].pop('recorded_vars')
                )
                perf['minute_perf'].update(perf['cumulative_risk_metrics'])

                daytime = perf['period_start'].strftime("%a")
                # Only analyze day is in the CustomBusinessDay of our calenday.
                if (daytime in workDayStrList):
                    minute_perfs.append(perf['minute_perf'])
            else:
                self.risk_report = perf`


`'minute_perf' in perf
Out[8]: 
False`

So, there is no minute_perf in perf varable. while, the daily_perf is in perf variable.

daily_perf in perf
Out[11]: 
True
```
due to  no minute_perf key in perf varable, the minute_stats is returned, it is a empty dataframe
```
`perf = zipline.run_algorithm(start=datetime(2018, 3, 8, 0, 0, 0, 0, pytz.utc),
                             end=datetime(2018, 3, 10, 0, 0, 0, 0, pytz.utc),
                             initialize=initialize,
                             trading_calendar=TwentyFourHR(),
                             capital_base=1000000,
                             handle_data=handle_data,
                             data_frequency='minute',
                             data=panel)
print(perf)
Empty DataFrame
Columns: []
Index: []`
```

I have download the files you uploaded,  replaced the algorithm file and run. it unable to run, 
some of packages are not able to import since i do not some files in my local envirenment. For example, from zipline.finance.performance import PerformanceTracker. the defult zipline 1.3.0 has no performance folder  in path\zipline\finance, it has no calendars folder in zipline\utils. As a result, I just revised the algorithm file by your description. it unable to show introday performance report.

[imkoukou]

@studyquant The file I posted are for a older version of Zipline. The Version 1.3.0 changed the folder structure, even the calendar has been separated as a single package folder outside the Zipline folder. I updated to Zipline 1.3.0 recently, and encountered the same problem at the first time, that 'minute_perf' not in perf. For me, the problem was solved by:

1. The calendar is default initialized with get_calendar("NYSE") in file of "zipline\utils\run_algo.py", that not match my minute data well. I customized it, and specified the customized calendar for minute frequency data.
2. I found that initialization of TradingAlgorithm class in file of "zipline\utils\run_algo.py",the emission_rate is not correctly specified even if I used a command --data-frequency minute to tell zipline run with minute mode. I added the parameter of "emission_rate = data_frequency".
3. Also as older version, added my function of "_create_minute_stats" to return minute_stats.

Make sure you have registered a suitable calendar for your minute csv data, including the date in the calendar.

That is all I've done for Zipline 1.3.0. And it can return the right minute data report to me.
algorithm for Zipline Version 1.3.0.zip

[studyquant]

@imkoukou
well, it works. However, I guess there are some error in algorithm file you provided. it shows this ...

  File "C:\Anaconda3.5-64\lib\site-packages\zipline\algorithm.py", line 759, in run
    for perf in self.get_generator():
  File "C:\Anaconda3.5-64\lib\site-packages\zipline\algorithm.py", line 632, in get_generator
    return self._create_generator(self.sim_params)
  File "C:\Anaconda3.5-64\lib\site-packages\zipline\algorithm.py", line 607, in _create_generator
    metrics_tracker.handle_start_of_simulation(benchmark_source)
  File "C:\Anaconda3.5-64\lib\site-packages\zipline\finance\metrics\tracker.py", line 144, in handle_start_of_simulation
    benchmark_source,
  File "C:\Anaconda3.5-64\lib\site-packages\zipline\finance\metrics\tracker.py", line 127, in hook_implementation
    impl(*args, **kwargs)
  File "C:\Anaconda3.5-64\lib\site-packages\zipline\finance\metrics\metric.py", line 190, in start_of_simulation
    daily_returns_series = benchmark_source.daily_returns(
AttributeError: 'NoneType' object has no attribute 'daily_returns'

Anyway, currently, the introday report is working. I really thanks for your help.
I have uploaded the algorithm I use below.
algorithm.zip

[imkoukou]

@studyquant The error may be caused some other changes I did to benchmark function, just ignore it. : )

[dpkdeepakpandey]

@studyquant @imkoukou I am new to zipline and currently facing an issue while returning the minute performance.I have a dataset that consist of minute by minute data of 1 week but while running the algorithm(zipline.run_algorithm) i am getting performance(perf) on daily basis.I had made the changes which need to be done in run_algo.py and algorithm.py but still the performance(perf) is reflecting on the basis of daily performance.The data is entering into _create_minute_stats but while returning it is giving empty dataframe.

[balut91]

@dpkdeepakpandey I found the solution.
In def _create_minute_stats(self, perfs):
comment if (daytime in workDayStrList):

The workDayStrList is coming some random value which is not correct. Anyways the working days is already taken care when generating the perf results. I don't think we need to put that condition again to check working day list.
Please reply to this if this is correct/wrong.

[dpkdeepakpandey]

Yeah workDayStrList was coming with value [1111100] which i guess was equivalent to [Mon,Tue,Wed,Thu,Fri,Sat,Sun] but it was not helping.So,for the same i have given by default my_working_ days as [Mon,Tue,Wed,Thu,Fri] and later on we are comparing this with daytime and it works fine for minute level data.

Currently i am facing issues when i want to do the same with '5min','10min','15min'.Every time it is expecting data as minute by minute. Does zipline support for these minute integration also as i am not fully aware of it.