How do I fetch incremental reports that return approved reports too within the date filters
I want to store & show report data for partner IDs in my application. I need to consider only
Approved/Reimbursed reports. Although this might change in the future. That is why I allow for reports with any status in my API request.
I implemented an incremental fetch strategy. So in the first fetch ever for a given partner ID, I fetch all the data from one year ago until that day. Post that, every 5 days, I pull the report again using the
inputSettings.filters.startDate and the
inputSettings.filters.endDate fields to limit the report to get only the new 5 days' worth of data. This makes sense in order to save unnecessary data requests.
Obviously, I get reports with various statuses like
Open/Closed/Processing in every periodic fetch that I don't care about yet. I get some
Approved/Reimbursed reports that I deem valid and need to show.
Here's the problem - the input date filters only limit reports based on when they were created/submitted. Not approved.
So assume that I fetch data from 1st January to 5th January, and get a report that's in the
Processing status. I don't care about it and don't consider it in my application. This report may be approved on 7th January. Then 5 days later, I fetch reports from 5th January to 10th January. But the report in question is not returned as I will still have to query 1st to 5th January to get that report's updated data again. I lose that piece of data.
Am I making any mistakes with the approach? Is there any alternative approach that I can use to counter this problem? Or should I pull all data from today back to 1 year ago at all times? That would require unique identifiers - to know which expense to update.
Any help would be greatly appreciated!