This could be a bit complicated, but I think something like the following might work.
Basically, every time motion is detected you would increase a counter by 1. If that counter is >=3 (ie this is the 3rd, 4th, 5th etc motion detected) then it should run some steps.
If not, it should wait 5 minutes before resetting the counter to 0. In the meantime, if motion is detected within the 5 minute window the counter will continue to increase until there have been enough triggers to qualify to run your steps.
- Trigger: Motion detected (I'm assuming via an IFTTT webhook?)
- Set variable motion_tracker + 1
- IF motion_tracker >= 3 THEN
- Run whatever steps should be triggered by motion
- Pause for 5 minutes
- Set variable motion_tracker = 0
- END IF
If that doesn't quite do what you need, you might try changing the step after the 5 minute pause and instead of resetting motion_tracker to 0 subtract 1 from it instead.