Heat olive oil in a skillet over medium heat; cook and stir until fat renders and bacon is slightly browned, 3 to 5 minutes. Add onion to bacon; cook and stir until soft and translucent, 3 to 5 minutes. Stir celery and garlic into onion mixture, cook until fragrant, 1 minute.
Pour clam juice, chicken broth, potato, cayenne pepper, and black pepper into bacon mixture; bring to a simmer, reduce heat to medium low, and cook until potatoes are almost tender, 12 to 15 minutes. Stir cream and chile pepper into potato mixture; bring to a simmer and cook until potatoes are completely tender and chowder flavors combine, about 5 minutes. Season with lemon zest and salt.
Stir scallops into chowder and cook over medium heat until scallops are tender and white, 1 to 3 minutes. Remove from heat; stir in tarragon.