How do I paginate results?

wonderingdev

Hi. My request to the API looks something like this -

My startDate is 01/01/2019. My approvedAfter is also 01/01/2019. Too many results end up timing out. How do I paginate over this?

Jeremy Boniface

@wonderingdev Are you providing a template in your request?

Jeremy Boniface

In my recommended setup you would not retarget the date range. You would continue to target the full date range until you've reached a range where expenses are aged out and submissions would no longer be approved.

wonderingdev

@Jeremy Boniface Yes I am. I am using addHeader as well.

Jeremy Boniface

Can you show the template code?

wonderingdev

Here's the template - https://pastebin.com/y1xa7L0F

Jeremy Boniface

Hm, the template compiles when I test it against a single test report. Can you run your API request against a small set of reports and confirm that it is returning results, so we can rule out any issues with the way it's being sent?

wonderingdev

@Jeremy Boniface Pardon me, I just read my question again and I get how it might have been misunderstood. The problem is not the template/request itself.

This request is being sent for a partner with a huge amount of reports. When I specify the date filters (startDate = 01/01/2019 & approvedAfter = 01/01/2019), the request times out due to the sheer number of reports that pass the filter condition. My question is if there is a way that I can paginate this data and get it in slices?

Jeremy Boniface

To clarify, I just wanted to make sure the timeout wasn't a 500 error or something of that nature. So just to be clear, you have validated that the request works for a small number of reports?

Regarding the pagination, I don't believe there is nor should there be a need for it, as far as I know, but perhaps we should get a better idea of just how many reports you're targeting. Do you have an estimate?

wonderingdev

@Jeremy Boniface Yes, I have verified that the request works for a small number of reports. If I can explain the need to do this in the first place -

I am trying to fetch all approved/reimbursed reports for the partner from 2019 and have to keep updating that list end of every week.

So for week 1 of 2019, I set the startDate and endDate filters to get all reports in week 1. In Week 2 I update the date filters to equal week 2 of 2019 and fetch data. But I keep missing reports that were created in week 1 of 2019 but approved in Week 2 of 2019.

Jeremy Boniface

Have you considered adding a label to the report on export via markedAsExported in onFinish, then filtering that label out in future requests?

In the template, the code is something like:

  <#assign exportedBefore = report.allExportsLabels?seq_contains("EXPORTED_VIA_API")>

  <#if !exportedBefore> print stuff</#if>

(or I think you can also use markedAsExported in the input settings instead of filtering in the template, as well)

I don't think this would solve the initial data dump, though, as you'd need to manually restrict the date range and send multiple requests if the problem is indeed a timeout, but it might help in the future (and help with keeping your data set clean).

I'm going to check with Engineering about the timeout issue, though.

wonderingdev

Thank you for the quick replies, @Jeremy Boniface!

Have you considered adding a label to the report on export via markedAsExported in onFinish, then filtering that label out in future requests?

I'm not sure if this would help.

Example to illustrate better - A report that is submitted on July 10th & approved on July 15th. If my first fetch uses startDate & endDate as July 4th - July 11th, I'll get a response with this report in the submitted status. I discard it because I only care about approved reports.

In my next fetch which uses startDate & endDate as July 12th - July 19th, I will not get this report in the approved status because this report was submitted out of the specified time period.

I feel like I am not using the filters correctly. Is there a way I can achieve this?

Jeremy Boniface

Confirmed with Engineering that the request would not time out on our side -- if it's timing out, it's on your side. There are no options for pagination, as addHeader handles streaming the data in chunks.

wonderingdev

Thanks @Jeremy Boniface!

wonderingdev

@Jeremy Boniface Can you please explain the following behavior for startDate & approvedAfter filters? I am not able to understand how this works.

For inputSettings.startDate & inputSettings.approvedAfter = 2019-01-01

1. When I set inputSettings.reportState to OPEN,SUBMITTED,APPROVED,REIMBURSED,ARCHIVED , I get ~16K approved reports.
2. When I set inputSettings.reportState to APPROVED,REIMBURSED , I get a different number of approved reports.

Jeremy Boniface

@wonderingdev Are you using the Approved (date) or Status column to count the reports? If you're using Approved, my guess would actually be that you're picking up ARCHIVED in your count, because CLOSED reports technically have an approval date in the data structure. In any case, I would recommend filtering to just ARCHIVED reports, and see if that matches your discrepancy.

wonderingdev

Thanks for the quick response. I am now finding consistent approved reports across fetches.

I am now getting consistent results no matter what the states are! 🤯 (it could be a bug but more often than not problems are user-generated!)

If I can bother you with one last thing - I just want to keep updating my list of approved/reimbursed reports. In that case, simply passing one filter, i.e. approvedAfter makes sense!

For example, on 16th July, I can query for all reports approvedAfter 2019-01-01. A week later, I can query for all reports approvedAfter 16th July.

This should help me capture ALL reports approved! But the API won't let me pass only one date filter, so I have to pass startDate which makes things a bit confusing for me. Can you confirm if there is an alternative to this?

Michelle Niemi

I do not believe there is an alternative to this. However, that would be a great question to ask your fellow API users within the Community as they may have found a way around that.