Location/Setting
Instances like from WoW or with DoD are not likely to be so good. But have say a case where you have "alerts" for signals/anomalies detected outside the normal habitable zone that you can go after and you can make those more replayable and non scripted. Make it so you go to one and you'll get one of a couple of random little maps/settings in an isolated area protected by your dropship or such, and then you deal with whatever is there. It might be a group of smaller enemies, or a single giant thing, or one sneaky stealthy thing. Perhaps have random modifiers that they might have or to the environment/weather to help keep things varied too. The idea is just that you can get some type of fight on demand (the upside of instances) but without it being the same thing over and over every time.
For the open world then you could have them wander in from the borders and attack bases, or just randomly wander depending on type/lore and be passive till a player attacks. Maybe have one that flies or just runs from one side of the instance to the other and players try to kill it before it escapes. You can also have them unborow or phase into the reality at semi random or in certain general locations or whatever, but avoid having it happen right on players already doing stuff if possible. Don't want to be doing a THMPR and bam, suddenly you have a extra boss wailing on your poor machine out of nowhere and making you lose rewards. Don't want anything like when the nados would spawn on a thumper in FF and you'd just lose everything do to a random sudden spawn that you can't predict or prepare for or counter.
Gameplay
I like games like Monster Hunter and Dark Souls and Shadow of the Colossus a lot. While obviously the gameplay will be much different with guns and jump jets and all some of the ideas are there. The idea that the enemy can throw out a couple of different types of attacks where you have to identify what it is based on the animation being used, the idea that it might be stationary or suddenly start moving, the idea that you might end up jetting/climbing onto it in order to attack certain weak spots... lots of things that make a giant weird creature more interesting to actually fight.
You can have attacks like something just slams the ground in front of it, maybe it does a giant leap and anyone standing where it targeted is in trouble, maybe it can breathe fire or lasers or shoot spikes out in a massive cone, all sorts of ways to threaten people be it up close or at a distance (preferably make it so you want to stay close to some types of enemies to avoid certain very dangerous ranged attacks). But don't make it so if you just get close and circle strafe behind it you're all good, let them slam their tail down to hit people right behind them quickly, let them bart, kick roks up behind them, turn their head for a quick laser/breath/howl, or let them suddenly do a big rearward leap and then do a close forward AOE attack so anyone behind them who didn't move quicklly gets hit. Lots of posobilities, I just don't like the idea of it being more like the Brontodon King where it's this massive thing that largley just stands there and walks/runs towards one person for an up close attack half the time before spinning on a dime to it's new target and then going after them. Make it so it's movement feels like it's own effort or series of actions if possible (again, like in the aforementioned games where it's a chance to reposition or attack or such).
Of coarse, depending on their net model this may not work to well to have the need for players to react to more immediate visual queues like this ( I know I couldn't play later Baneclaws reliably since I'd get hit by the little claws that I had dodged all the time, while often having others roll right through me and do nothing. If it's that fast paced or limited like that then a lot of this won't work).
Rewards
Some types of random drop rewards are ok. Just keep them from being power stuff, or permanent pieces of accomplishment (like BBB). Say you can randomly get the head of the boss and pin it on a banner pole over the back of your frame, slowly dripping blood before it fades an hour after activation (and can't be stored for more than one week in inventory either). Having fun/interesting things like that can be nice. And if it's a low chance limited duration thing then you won't ever have the entire world full of players wearing them so they remain interesting things that newer players will see and be curious about. Also just fun to have a little something left over every so often from something you did, just make sure it's not something you feel you really miss out on if you don't end up getting it.
You could also do a limited type of data tokens for a boss where each kill you make gives you a data token on it and after five or so of those you unlock the ability to idle/project a hologram of that boss/creature. Maybe if there is an in game encyclopedia on stuff put an entry to read about it there. Just make sure it's a small enough number that a person gets it quickly enough so it's not like in warframe where you stop and snap pictures of 200 or something targets to get an unlock. And jsut being a kill reward it won't interfere with players gameplay like those cameras did.
Another idea is a picture reward. If players can have bases, or archive other stuff you could make it so after a big boss kill a player can use a camera bot or something you buy near the corpse to take a picture of it. This will show all the other players standing with it too. These pics can be linked to and sent in game or just posted on the wall of an army/groups base or such if they have space. Or rather than linking you could have characters project saved images/archived stuff like this as a hologram so other people around them can see it in game (better than a mail that they have to open a UI to look at since that breaks immersion and game play flow and isn't as easy/intuitive to show a group something).
Then there's all the other more general ideas that can be given every time like resources, relic parts (combine three to get a choice of different artifacts/equipment/resource), bounty money, possibly animal parts if they can be used for anything (some of these guys have cybernetics implanted in them, and who knows what their bones/scales/hair could be used for... maybe we can make bad ass cloaks for our suits(with holes for jets)). Preferably nothing that a person will need on a regular basis or in particularly large amounts of for anything, just some nice crafting mats for the odd custom cloak/armor/item that they might want when they want a new look. Again, don't make it something to grind over.
EDIT: Some boss ideas.
Giant flying creature that sails around raining balls of fiery/plasmaie death onto people/bases.
- -Mostly just something players shoot at from the ground, it will rarely stay right over one area too long as it prefers to circle or fly out to other areas (It does not hover, it constantly has to be moving/flying).
- -If you do have a fast jet potent frame and can get up to it you can interact to do a shadow of the colossus style melee/cinematic gun attack on it.
- -Will try to run away after it takes too much damage, but it's flight speed will drop off after short bursts so players will be able to catch back up after it moves the area of combat.
Giant bipedal thing.
- -Does massive leaps to get through he air from location to location, relatively agile, smashes the ground with fists and stomps and such. Also has fire breath or something.
Robot swarm, a swarm of player sized robots that build something really big.
- -The giant robot has big ranged attacks mostly, very dangerous but also vulnerable to poke as it's individual parts easily get shot off.
- -Once it loses to many smaller parts (just break them off with % of health lost or maybe by individual health hit boxes) it breaks apart and now you have a swarm of these smaller machines flooding over the map towards players. They can creep up cliff walls, fly, and fire small particle guns that are semi slow projectiles. In this case they can move very fast and swarm players. Once you kill enough of the swarm they reform into a second stage of the the boss that's a smaller robot form.
- -Rinse repeat for possibly a third stage.
Invisible hunter, probably humanoid with a tattered cloak covering most of it's body. Invisible or only seen as a slight shimmer where it appears.
- -Can, to a limited extent teleport while cloaked.
- -Not super aggressive, this guy likes to disappear and will wait till he can get a good hit in when possible.
- -You start to see him as a shimmer if he gets relativly close to you, so if he's coming to stab you you can likely see him coming, can't teleport to within 20 meters of a player by the way.
- -His attacks do disrupt his stealth to a fairly large extent. You can also make it get disrupted and shimmer (to all players regardless of range) by attacking him.
- -Relatively low health, and not much bigger than an omniframe. Very different pacing and feel from larger brutal bosses.
- -A great boss to spawn when a swarm of enemy units attacks a base, since he can try to get inside and damage defenses or shield gens or stuff.
Long legged augmented/deformed Pangolin like thing.
- -can shoot spikes/scales at players all around it.
- -Can curl into a ball and roll around, invincible while rolling but will uncurl if he can't hit anyone for awhile (time to jet)
- -Can dig underground/burrow. Then can burst up elsewhere near some players.
- -Doesn't have a lot of health like the giant bosses, but does have a decent number of invuln/dmg reduction phases between rolling and digging so you want to bait his normal attacks if you can.